2019

Non-Gaussianity of Stochastic Gradient Noise

Panigrahi, Abhishek, Somani, Raghav, Goyal, Navin et al.

Understand

What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention.

In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training.
We observe that for batch sizes 256 and above, the distribution is best described as Gaussian at-least in the early phases of training.
This holds across data-sets, architectures, and other choices.

Built on

The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects
Original
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma · 1910
Earlier work this paper cites.
A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
A test of goodness of fit
T. W. Anderson and D. A. Darling · 1954
Earlier work this paper cites.
An analysis of variance test for normality (complete samples)
S. S. Shapiro and M. B. Wilk · 1965
Earlier work this paper cites.
Comparison of learning algorithms for handwritten digit recognition
Yann LeCun, LD Jackel, Leon Bottou, A Brunot, Corinna Cortes, JS Denker, Harris Drucker, I Guyon, UA Muller, Eduard Sackinger, et al · 1995
Earlier work this paper cites.
Langevin diffusions and metropolis-hastings algorithms
Gareth O Roberts and Osnat Stramer · 2002
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.

Similar

Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests
Nornadiah Mohd Razali and Yap Bee Wah · 2011
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Cited alongside, same era.
On estimating the tail index and the spectral measure of multivariate α \alpha -stable distributions
Mohammad Mohammadi, Adel Mohammadpour, and Hiroaki Ogata · 2015
Cited alongside, same era.
Deep learning
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
A variational analysis of stochastic gradient algorithms
Stephan Mandt, Matthew Hoffman, and David Blei · 2016
Cited alongside, same era.
On the diffusion approximation of nonconvex stochastic gradient descent
Original
Wenqing Hu, Chris Junchi Li, Lei Li, and Jian-Guo Liu · 2017
Cited alongside, same era.

Then

Three factors influencing minima in sgd
Original
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey · 2017
Later among the works it cites.
Stochastic modified equations and adaptive stochastic gradient algorithms
Qianxiao Li, Cheng Tai, et al · 2017
Later among the works it cites.
Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis
Original
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky · 2017
Later among the works it cites.
A hitting time analysis of stochastic gradient langevin dynamics
Yuchen Zhang, Percy Liang, and Moses Charikar · 2017
Later among the works it cites.
Optimization methods for large-scale machine learning
L. Bottou, F. Curtis, and J. Nocedal · 2018
Later among the works it cites.
Stable Distributions - Models for Heavy Tailed Data
J. P. Nolan · 2018
Later among the works it cites.
High-Dimensional Probability
Roman Vershynin · 2018
Later among the works it cites.
A tail-index analysis of stochastic gradient noise in deep neural networks
Umut Simsekli, Levent Sagun, and Mert Gürbüzbalaban · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…