2019

Implicit Regularization in Over-parameterized Neural Networks

Kubo, Masayoshi, Banno, Ryotaro, Manabe, Hidetaka et al.

Understand

Over-parameterized neural networks generalize well in practice without any explicit regularization.

Although it has not been proven yet, empirical evidence suggests that implicit regularization plays a crucial role in deep learning and prevents the network from overfitting.
In this work, we introduce the gradient gap deviation and the gradient deflection as statistical measures corresponding to the network curvature and the Hessian matrix to analyze variations of network derivatives with respect to input parameters, and investigate how implicit regularization works in ReLU neural networks from both theoretical and empirical perspectives.
Our result reveals that the network output between each pair of input samples is properly controlled by random initialization and stochastic gradient descent to keep interpolating between samples almost straight, which results in low complexity of over-parameterized neural networks.

Built on

An invariance principle for conditioned sums of independent random variables
Thomas M Liggett · 1968
Earlier work this paper cites.
Bayesian Learning for Neural Networks
Radford M. Neal · 1994
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al · 1998
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Deep learning via hessian-free optimization
James Martens · 2010
Earlier work this paper cites.
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton · 2013
Earlier work this paper cites.
On the number of linear regions of deep neural networks
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
In search of the real inductive bias: On the role of implicit regularization in deep learning
Original
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Random walk initialization for training very deep feedforward networks
Original
David Sussillo and LF Abbott · 2014
Earlier work this paper cites.
Train faster, generalize better: Stability of stochastic gradient descent
Original
Moritz Hardt, Benjamin Recht, and Yoram Singer · 2015
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
All you need is a good init
Original
Dmytro Mishkin and Jiri Matas · 2015
Earlier work this paper cites.
On the expressive power of deep learning: A tensor analysis
Nadav Cohen, Or Sharir, and Amnon Shashua · 2016
Earlier work this paper cites.
Deep neural networks with random gaussian weights: A universal classification strategy?
Raja Giryes, Guillermo Sapiro, and Alexander M Bronstein · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Identity mappings in deep residual networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
On large-batch training for deep learning: Generalization gap and sharp minima
Original
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang · 2016
Earlier work this paper cites.

Similar

Exponential expressivity in deep neural networks through transient chaos
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli · 2016
Cited alongside, same era.
On the expressive power of deep neural networks
Original
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein · 2016
Cited alongside, same era.
Benefits of depth in neural networks
Original
Matus Telgarsky · 2016
Cited alongside, same era.
Diverse neural network learns true target functions
Original
Bo Xie, Yingyu Liang, and Le Song · 2016
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Cited alongside, same era.
A closer look at memorization in deep networks
Original
Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al · 2017
Cited alongside, same era.
The shattered gradients problem: If resnets are the answer, then what is the question?
Original
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams · 2017
Cited alongside, same era.
Spectrally-normalized margin bounds for neural networks
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky · 2017
Cited alongside, same era.
Sgd learns over-parameterized networks that provably generalize on linearly separable data
Original
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz · 2017
Cited alongside, same era.
Sharp minima can generalize for deep nets
Original
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio · 2017
Cited alongside, same era.
Deep neural networks as gaussian processes
Original
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2017
Cited alongside, same era.
The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning
Original
Siyuan Ma, Raef Bassily, and Mikhail Belkin · 2017
Cited alongside, same era.

Then

Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang · 2018
Later among the works it cites.
A convergence theory for deep learning via over-parameterization
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Later among the works it cites.
On the convergence rate of training recurrent neural networks
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Later among the works it cites.
Stronger generalization bounds for deep nets via a compression approach
Original
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang · 2018
Later among the works it cites.
Does data interpolation contradict statistical optimality?
Original
Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov · 2018
Later among the works it cites.
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
Pratik Chaudhari and Stefano Soatto · 2018
Later among the works it cites.
Gradient descent provably optimizes over-parameterized neural networks
Original
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh · 2018
Later among the works it cites.
Learning overparameterized neural networks via stochastic gradient descent on structured data
Yuanzhi Li and Yingyu Liang · 2018
Later among the works it cites.
A surprising linear relationship predicts test performance in deep networks
Original
Qianli Liao, Brando Miranda, Andrzej Banburski, Jack Hidary, and Tomaso Poggio · 2018
Later among the works it cites.
Gradient descent quantizes relu network features
Original
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly · 2018
Later among the works it cites.
Gaussian process behaviour in wide deep neural networks
Original
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani · 2018
Later among the works it cites.
Sensitivity and generalization in neural networks: an empirical study
Original
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2018
Later among the works it cites.
Deep learning generalizes because the parameter-function map is biased towards simple functions
Original
Guillermo Valle Pérez, Ard A Louis, and Chico Q Camargo · 2018
Later among the works it cites.
Theory of deep learning iii: explaining the non-overfitting puzzle
Original
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar · 2018
Later among the works it cites.
Theory iiib: Generalization in deep networks
Original
Tomaso Poggio, Qianli Liao, Brando Miranda, Andrzej Banburski, Xavier Boix, and Jack Hidary · 2018
Later among the works it cites.
The implicit bias of gradient descent on separable data
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro · 2018
Later among the works it cites.
Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks
Original
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington · 2018
Later among the works it cites.
Fixup initialization: Residual learning without normalization
Original
Hongyi Zhang, Yann Dauphin, and Tengyu Ma · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…