2018

Fluctuation-dissipation relations for stochastic gradient descent

Yaida, Sho

Understand

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.

In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.
Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.
These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.

Built on

Über die von der molekularkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen
Albert Einstein · 1905
Earlier work this paper cites.
Zur kinetischen theorie der brownschen molekularbewegung und der suspensionen
Marian Von Smoluchowski · 1906
Earlier work this paper cites.
Reciprocal relations in irreversible processes. I
Lars Onsager · 1931
Earlier work this paper cites.
A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
Markoff random processes and the statistical mechanics of time-dependent phenomena. II. Irreversible processes in fluids
Melville S Green · 1954
Earlier work this paper cites.
Statistical-mechanical theory of irreversible processes. I. General theory and simple applications to magnetic and conduction problems
Ryogo Kubo · 1957
Earlier work this paper cites.
The Fokker-Planck equation: methods of solution and applications
Hannes Risken · 1984
Earlier work this paper cites.
Fokker-Planck description of learning in backpropagation networks
Günter Radons, Heinz Georg Schuster, and D Werner · 1990
Earlier work this paper cites.
Stochastic gradient learning in neural networks
Léon Bottou · 1991
Earlier work this paper cites.
Stochastic processes in physics and chemistry , volume 1
Nicolaas Godfried Van Kampen · 1992
Earlier work this paper cites.
Weight space probability densities in stochastic learning: I. Dynamics and equilibria
Todd K Leen and John E Moody · 1993
Earlier work this paper cites.
Nonequilibrium equality for free energy differences
Christopher Jarzynski · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.

Similar

Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences
Gavin E Crooks · 1999
Cited alongside, same era.
Absence of thermodynamic phase transition in a model glass former
Ludger Santen and Werner Krauth · 2000
Cited alongside, same era.
Stochastic methods , volume 4
Crispin Gardiner · 2009
Cited alongside, same era.
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton · 2009
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Cited alongside, same era.
Equilibrium equation of state of a hard sphere binary mixture at very large densities using replica exchange Monte Carlo simulations
Gerardo Odriozola and Ludovic Berthier · 2011
Cited alongside, same era.
Bayesian learning via stochastic gradient Langevin dynamics
Max Welling and Yee W Teh · 2011
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
In search of the real inductive bias: On the role of implicit regularization in deep learning
Original
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2014
Cited alongside, same era.
Stochastic modified equations and adaptive stochastic gradient algorithms
Original
Qianxiao Li, Cheng Tai, and Weinan E · 2015
Cited alongside, same era.
Continuous-time limit of stochastic gradient descent revisited
Stephan Mandt, Matthew D Hoffman, and David M Blei · 2015
Cited alongside, same era.
Consistency and fluctuations for stochastic gradient Langevin dynamics
Yee Whye Teh, Alexandre H Thiery, and Sebastian J Vollmer · 2016
Cited alongside, same era.

Then

Three factors influencing minima in SGD
Original
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey · 2017
Later among the works it cites.
Batch size matters: A diffusion approximation framework on nonconvex stochastic gradient descent
Original
Chris Junchi Li, Lei Li, Junyang Qian, and Jian-Guo Liu · 2017
Later among the works it cites.
Stochastic gradient descent as approximate Bayesian inference
Stephan Mandt, Matthew D Hoffman, and David M Blei · 2017
Later among the works it cites.
Natural Langevin dynamics for neural networks
Gaétan Marceau-Caron and Yann Ollivier · 2017
Later among the works it cites.
Exploring generalization in deep learning
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro · 2017
Later among the works it cites.
Stochastic modified equations for the asynchronous stochastic gradient descent
Original
Jing An, Jianfeng Lu, and Lexing Ying · 2018
Closest in time.
Optimization methods for large-scale machine learning
Léon Bottou, Frank E Curtis, and Jorge Nocedal · 2018
Closest in time.
On the convergence of Adam and beyond
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar · 2018
Closest in time.
ConvNetJS CIFAR-10 demo
Andrej Karpathy · 2018
Closest in time.
Stochastic gradient Langevin dynamics that exploit neural network structure, 2018
Zachary Nado, Jasper Snoek, Roger Grosse, David Duvenaud, Bowen Xu, and James Martens · 2018
Closest in time.
A Bayesian perspective on generalization and stochastic gradient descent
Original
Samuel L Smith and Quoc V Le · 2018
Closest in time.
The implicit bias of gradient descent on separable data
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro · 2018
Closest in time.
The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects
Original
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…