2018

Layer rotation: a surprisingly powerful indicator of generalization in deep networks?

Carbonnelle, Simon, De Vleeschouwer, Christophe

Understand

Our work presents extensive empirical evidence that layer rotation, i.e.

the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.
In particular, larger cosine distances between final and initial weights of each layer consistently translate into better generalization performance of the final model.
Interestingly, this relation admits a network independent optimum: training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 30% test accuracy.

Built on

Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, and Paolo Frasconi · 1994
Earlier work this paper cites.
Flat minima
Sepp Hochreiter and Jurgen Schmidhuber · 1997
Earlier work this paper cites.
The Sample Complexity of Pattern Classification with Neural Networks : The Size of the Weights is More Important than the Size of the Network
Peter L Bartlett · 1998
Earlier work this paper cites.
The vanishing gradient problem during learning recurrent neural nets and problem solutions
Sepp Hochreiter · 1998
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Optimization On Manifolds : Methods And Applications
P.-A. Absil, R. Mahony, and R. Sepulchre · 2010
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012
Tijmen Tieleman and Geoffrey Hinton · 2012
Earlier work this paper cites.
Robustness and generalization
Huan Xu and Shie Mannor · 2012
Earlier work this paper cites.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Keras, 2015
François Chollet et al · 2015
Earlier work this paper cites.
Qualitatively characterizing neural network optimization problems
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Lei Ba · 2015
Earlier work this paper cites.

Similar

In search of the real inductive bias: On the role of implicit regularization in deep learning
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2015
Cited alongside, same era.
Norm-Based Capacity Control in Neural Networks
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2015
Cited alongside, same era.
92.45% on CIFAR-10 in Torch
Sergey Zagoruyko · 2015
Cited alongside, same era.
TensorFlow : Large-Scale Machine Learning on Heterogeneous Distributed Systems
Original
Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Vi, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2016
Cited alongside, same era.
Tiny ImageNet Visual Recognition Challenge
Stanford CS231N · 2016
Cited alongside, same era.
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Wide Residual Networks
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Train longer , generalize better : closing the generalization gap in large batch training of neural networks
Elad Hoffer, Itay Hubara, and Daniel Soudry · 2017
Cited alongside, same era.
Three Factors Influencing Minima in SGD
Original
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey · 2017
Cited alongside, same era.
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang · 2017
Cited alongside, same era.
Exploring Generalization in Deep Learning
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nathan Srebro · 2017
Cited alongside, same era.
Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates
Original
Leslie N Smith and Nicholay Topin · 2017
Cited alongside, same era.

Then

Normalized gradient with adaptive stepsize method for deep neural network training
Original
Adams Wei Yu, Qihang Lin, Ruslan Salakhutdinov, and Jaime Carbonell · 2017
Later among the works it cites.
Understanding deep learning requires re-thinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2017
Later among the works it cites.
Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling, 2018
Boris Ginsburg, Igor Gitman, and Yang You · 2018
Closest in time.
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Original
Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He · 2018
Closest in time.
Norm matters: efficient and accurate normalization schemes in deep networks
Original
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry · 2018
Closest in time.
Revisiting Small Batch Training for Deep Neural Networks
Original
Dominic Masters and Carlo Luschi · 2018
Closest in time.
Insights on representational similarity in neural networks with canonical correlation
Ari Morcos, Maithra Raghu, and Samy Bengio · 2018
Closest in time.
On the importance of single directions for generalization
Ari S Morcos, David G T Barrett, Neil C Rabinowitz, and Matthew Botvinick · 2018
Closest in time.
Sensitivity and generalization in neural networks: an empirical study
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein · 2018
Closest in time.
A Walk with SGD
Original
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio · 2018
Closest in time.
Stiffness: A New Perspective on Generalization in Neural Networks
Stanislav Fort, Paweł Krzysztof, and Nowak Srini · 2019
Closest in time.
On the relation between the sharpest directions of DNN loss and the SGD step length
Stanisław Jastrzȩbski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey · 2019
Closest in time.
Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes · 2019
Closest in time.
Three mechanisms of weight decay regularization
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…