2019

Mean Field Analysis of Deep Neural Networks

Sirignano, Justin, Spiliopoulos, Konstantinos

Understand

We analyze multi-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations.

We rigorously establish the limiting behavior of the multi-layer neural network output.
The limit procedure is valid for any number of hidden layers and it naturally also describes the limiting behavior of the training loss.
The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization.

Built on

A mean field theory of two-layers neural networks: dimension free bounds and kernel limit
Original
S. Mei, T. Misiakiewicz and A. Montanari · 1902
Earlier work this paper cites.
Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
Original
P.-M. Nguyen · 1902
Earlier work this paper cites.
Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
K. Hu, Z. Ren, D. Šiška, and L. Szpruch · 1905
Earlier work this paper cites.
A mean-field limit for certain deep neural networks
D. Araújo, R. I. Oliveira and D. Yukimura · 1906
Earlier work this paper cites.
Sparse Optimization on Measures with Over-parameterized Gradient Descent
L. Chizat · 1907
Earlier work this paper cites.
D.A. Dawson and K.J. Hochberg Wandering random measures in the Fleming-Viot model, Annals of Probability 10 (1982), pp. 554-580
1982
Earlier work this paper cites.
Nonlinear reflecting diffusion processes, and the propagation of chaos and fluctuations associated
A-S. Sznitman · 1984
Earlier work this paper cites.
Markov Processes: Characterization and Convergence
S. Ethier and T. Kurtz · 1986
Earlier work this paper cites.
Approximation by superposition of a sigmoidal function
G. Cybenko · 1989
Earlier work this paper cites.
On the approximate realization of continuous mappings by neural networks
Ken-Ichi Funahashi · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
K. Hornik, M. Stinchcombe, and H. White · 1989
Earlier work this paper cites.
D.A. Dawson and K.J. Hochberg, A multilevel branching model, Advances in Applied Probability 23, (1991), pp. 701-715
1991
Earlier work this paper cites.
Approximation capabilities of multilayer feedforward networks
K. Hornik · 1991
Earlier work this paper cites.
Topics in propagation of chaos
A-S. Sznitman · 1991
Earlier work this paper cites.
D.A. Dawson and A. Greven, Hierarchically interacting Fleming-Viot processes with selection and mutation: multiple space-time scale analysis and quasi equilibria, Electronic J. of Prob., vol. 4, paper 4, (1993), pp. 1-81
1993
Earlier work this paper cites.
Approximation and estimation bounds for artificial neural networks
A. Barron · 1994
Earlier work this paper cites.
D.A. Dawson, K.J. Hochberg and V. Vinogradov, High-density limits of hierarchically structured branching-diﬀusing populations, Stochastic Processes and their Applications, 62, (1996), pp. 191-222
1996
Earlier work this paper cites.
D.A. Dawson and Y. Wu, Multilevel multitype models of an information system, I.M.A. Volume 84, Classical and Modern Branching Processes, eds. K.B. Athreya and P. Jagers, Springer-Verlag, (1996), pp. 57-72
1996
Earlier work this paper cites.
D.A. Dawson, Hierarchical and mean-ﬁeld stepping stone models, in Progress in Population Genetics and Human Evolution, eds. P. Donnelly and S. Tavare, Springer, IMA Volume in Mathematics and its Applications vol. 87. (1997)
1997
Earlier work this paper cites.
D.A. Dawson and J. Gärtner, Analytic aspects of multilevel large deviations, in Asymptotic Methods in Probability and Statistics (ed. B. Szyszkowicz), Elsevier, Amsterdam, (1998), pp. 401-440
1998
Earlier work this paper cites.
A Primer of Real Analytic Functions, second edition
S.G. Krantz and H.R. Parks · 2002
Earlier work this paper cites.

Similar

Gradient flows: in metric spaces and in the space of probability measures
L. Ambrosio, N. Gigli, and G. Savaré · 2008
Cited alongside, same era.
Learning Multiple Layers of Features from Tiny Images,
A. Krizhevsky · 2009
Cited alongside, same era.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010
2010
Cited alongside, same era.
Nonlinear Markov processes and kinetic equations
V.N. Kolokoltsov · 2010
Cited alongside, same era.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. Le · 2014
Cited alongside, same era.
Deepface: Closing the gap to human-level performance in face verification
Y. Taigman, M. Yang, M. Ranzato, L. Wolf · 2014
Cited alongside, same era.
Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning
B. Alipanahi, A. Delong, M. Weirauch, and B. Frey · 2015
Cited alongside, same era.
Deep Learning
Y. LeCun, Y. Bengio, and G. Hinton · 2015
Cited alongside, same era.
End to end learning for self-driving cars,
Original
M. Bojarski, D. Del Test, D. Dworakowski, B. Firnier, B. Flepp, P. Goyal, L. Jackel, M. Monfort, U. Muller, J. Zhang, and X. Zhang · 2016
Cited alongside, same era.
A family of functional inequalities, Łojasiewicz inequalities and displacement convex functions
Original
A. Blanchet and J. Bolte · 2016
Cited alongside, same era.
Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Cited alongside, same era.
Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
J. Ling, A. Kurzawski, and J. Templeton · 2016
Cited alongside, same era.

Then

Very deep convolutional networks for end-to-end speech recognition
Y. Zhang, W. Chan, and N. Jaitly · 2017
Later among the works it cites.
Scaling limit: Exact and tractable analysis of online learning algorithms with applications to regularized regression and PCA
Original
C. Wang, J. Mattingly, and Y. Lu · 2017
Later among the works it cites.
A note on lazy training in supervised differentiable programming
Original
L. Chizat, and F. Bach · 2018
Later among the works it cites.
On the global convergence of gradient descent for over-parameterized models using optimal transport
L. Chizat, and F. Bach · 2018
Later among the works it cites.
D.A. Dawson, Multilevel mutation-selection systems and set-valued duals, Journal of Mathematical Biology, Vol. 76, Issue 1–2, (2018), pp 295–378
2018
Later among the works it cites.
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
A. Jacot, F. Gabriel, and C. Hongler · 2018
Later among the works it cites.
Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone
Y. Leviathan and Y. Matias · 2018
Later among the works it cites.
Over-Parameterized Deep Neural Networks Have No Strict Local Minima For Any Continuous Activations
D. Li, T. Ding, and R. Sun · 2018
Later among the works it cites.
Adding One Neuron Can Eliminate All Bad Local Minima
S. Liang, R. Sun, J. Lee, and R. Srikant · 2018
Later among the works it cites.
Understanding the Loss Surface of Neural Networks for Binary Classification
S. Liang, R. Sun, Y. Li, and R. Srikant · 2018
Later among the works it cites.
A mean field view of the landscape of two-layer neural networks
S. Mei, A. Montanari, and P. Nguyen · 2018
Later among the works it cites.
Neural Networks as Interacting Particle Systems: Asymptotic Convexity of the Loss Landscape and Universal Scaling of the Approximation Error
Original
G. M. Rotskoff and E. Vanden-Eijnden · 2018
Later among the works it cites.
Universal features of price formation in financial markets: perspectives from Deep Learning
Original
J. Sirignano and R. Cont · 2018
Later among the works it cites.
J. Sirignano and K. Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, Journal of Computational Physics
2018
Later among the works it cites.
The limits and potentials of deep learning for robotics
N. Sunderhauf, O. Brock, W. Cheirer, R. Hadsell, D. Fox, J. Leitner, B. Upcroft, P. Abbeel, W. Burgard, M. Milford, and P. Corke · 2018
Later among the works it cites.
Gradient Descent Finds Global Minima of Deep Neural Networks
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai · 2019
Closest in time.
Gradient Descent Provably Optimizes Over-parameterized Neural Networks
S. Du, X. Zhai, B. Poczos, and A. Singh · 2019
Closest in time.
Kurdyka-Łojasiewicz-Simon inequality for gradient flows in metric spaces
D. Hauer and J. Mazón · 2019
Closest in time.
Mean Field Analysis of Neural Networks: A Law of Large Numbers
J. Sirignano and K. Spiliopoulos · 2020
Closest in time.
Mean Field Analysis of Neural Networks: A central limit theorem
J. Sirignano and K. Spiliopoulos · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…