2018

Greedy Layerwise Learning Can Scale to ImageNet

Belilovsky, Eugene, Eickenberg, Michael, Oyallon, Edouard

Understand

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power.

Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks.
Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success.
We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset.

Built on

Cybernetic predicting devices
Ivakhnenko, A. G. and Lapa, V · 1965
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
Cybenko, G · 1989
Earlier work this paper cites.
Optimal brain damage
LeCun, Y., Denker, J. S., and Solla, S. A · 1990
Earlier work this paper cites.
Approximation and estimation bounds for artificial neural networks
Barron, A. R · 1994
Earlier work this paper cites.
Training mlps layer by layer using an objective function for internal representations
Lengellé, R. and Denoeux, T · 1996
Earlier work this paper cites.
Approximation theory of the mlp model in neural networks
Pinkus, A · 1999
Earlier work this paper cites.
Convex neural networks
Bengio, Y., Roux, N. L., Vincent, P., Delalleau, O., and Marcotte, P · 2006
Earlier work this paper cites.
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., and Teh, Y.-W · 2006
Earlier work this paper cites.
Greedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H · 2007
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Image classification with the fisher vector: Theory and practice
Sánchez, J., Perronnin, F., Mensink, T., and Verbeek, J · 2013
Earlier work this paper cites.
Provable bounds for learning some deep representations
Arora, S., Bhaskara, A., Ge, R., and Ma, T · 2014
Earlier work this paper cites.
Breaking the curse of dimensionality with convex neural networks
Original
Bach, F · 2014
Earlier work this paper cites.
Dark knowledge
Hinton, G., Vinyals, O., and Dean, J · 2014
Earlier work this paper cites.
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Zeiler, M. D. and Fergus, R · 2014
Earlier work this paper cites.
Beating the perils of non-convexity: Guaranteed training of neural networks using tensor methods
Original
Janzamin, M., Sedghi, H., and Anandkumar, A · 2015
Earlier work this paper cites.
Deeply-supervised nets
Lee, C.-Y., Xie, S., Gallagher, P., Zhang, Z., and Tu, Z · 2015
Earlier work this paper cites.
Deep roto-translation scattering for object classification
Oyallon, E. and Mallat, S · 2015
Earlier work this paper cites.

Similar

ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2015
Cited alongside, same era.
Understanding neural networks through deep visualization
Original
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H · 2015
Cited alongside, same era.
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., and Vinyals, O · 2016
Cited alongside, same era.
Adanet: Adaptive structural learning of artificial neural networks
Original
Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M., and Yang, S · 2016
Cited alongside, same era.
Highway and residual networks learn unrolled iterative estimation
Original
Greff, K., Srivastava, R. K., and Schmidhuber, J · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Decoupled neural interfaces using synthetic gradients
Original
Jaderberg, M., Czarnecki, W. M., Osindero, S., Vinyals, O., Graves, A., Silver, D., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Random synaptic feedback weights support error backpropagation for deep learning
Lillicrap, T. P., Cownden, D., Tweed, D. B., and Akerman, C. J · 2016
Cited alongside, same era.
Understanding deep convolutional networks
Mallat, S · 2016
Cited alongside, same era.
Pruning convolutional neural networks for resource efficient inference
Original
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J · 2016
Cited alongside, same era.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.
Freezeout: Accelerate training by progressively freezing layers
Original
Brock, A., Lim, T., Ritchie, J., and Weston, N · 2017
Cited alongside, same era.

Then

Deep incremental boosting
Original
Mosca, A. and Magoulas, G. D · 2017
Later among the works it cites.
Building a regular decision boundary with deep networks
Oyallon, E · 2017
Later among the works it cites.
Evolution strategies as a scalable alternative to reinforcement learning
Original
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I · 2017
Later among the works it cites.
Mastering the game of go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al · 2017
Later among the works it cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Later among the works it cites.
Deep growing learning
Wang, G., Xie, X., Lai, J., and Zhuo, J · 2017
Later among the works it cites.
Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Allen-Zhu, Z., Li, Y., and Liang, Y · 2018
Closest in time.
Understanding deep neural networks with rectified linear units
Arora, R., Basu, A., Mianjy, P., and Mukherjee, A · 2018
Closest in time.
Assessing the scalability of biologically-motivated deep learning algorithms and architectures
Original
Bartunov, S., Santoro, A., Richards, B. A., Hinton, G. E., and Lillicrap, T · 2018
Closest in time.
Improved learning of one-hidden-layer convolutional neural networks with overlaps
Original
Du, S. S. and Goel, S · 2018
Closest in time.
i-revnet: Deep invertible networks
Jacobsen, J.-H., Smeulders, A., and Oyallon, E · 2018
Closest in time.
A provably correct algorithm for deep learning that actually works
Original
Malach, E. and Shalev-Shwartz, S · 2018
Closest in time.
Deep cascade learning
Marquez, E. S., Hare, J. S., and Niranjan, M · 2018
Closest in time.
Towards understanding the role of over-parametrization in generalization of neural networks
Original
Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., and Srebro, N · 2018
Closest in time.
Scattering networks for hybrid representation learning
Oyallon, E., Zagoruyko, S., Huang, G., Komodakis, N., Lacoste-Julien, S., Blaschko, M. B., and Belilovsky, E · 2018
Closest in time.
Neural networks with finite intrinsic dimension have no spurious valleys
Original
Venturi, L., Bandeira, A., and Bruna, J · 2018
Closest in time.
The effectiveness of layer-by-layer training using the information bottleneck principle, 2019
Elad, A., Haviv, D., Blau, Y., and Michaeli, T · 2019
Closest in time.
Biologically-plausible learning algorithms can scale to large datasets
Xiao, W., Chen, H., Liao, Q., and Poggio, T. A · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…