2019

Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Nguyen, Phan-Minh

Understand

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large.

We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks.
While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

Built on

Yoshua Bengio, Nicolas L Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte, Convex neural networks , Advances in neural information processing systems, 2006, pp. 123–130
2006
Earlier work this paper cites.
Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma, Provable bounds for learning some deep representations , International Conference on Machine Learning, 2014, pp. 584–592
2014
Earlier work this paper cites.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun, The loss surfaces of multilayer networks , Artificial Intelligence and Statistics, 2015, pp. 192–204
2015
Earlier work this paper cites.
Tamir Hazan and Tommi Jaakkola, Steps toward deep kernel methods from infinite neural networks , arXiv preprint arXiv:1508.05133 (2015)
Original
2015
Earlier work this paper cites.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, Deep learning , nature 521
2015
Earlier work this paper cites.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Fei-Fei Li, Imagenet large scale visual recognition challenge , International Journal of Computer Vision 115
2015
Earlier work this paper cites.
Karen Simonyan and Andrew Zisserman, Very deep convolutional networks for large-scale image recognition , International Conference on Learning Representations (2015)
2015
Earlier work this paper cites.
Hrushikesh N Mhaskar and Tomaso Poggio, Deep vs. shallow networks: An approximation theory perspective , Analysis and Applications 14
2016
Earlier work this paper cites.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli, Exponential expressivity in deep neural networks through transient chaos , Advances in neural information processing systems, 2016, pp. 3360–3368
2016
Earlier work this paper cites.
Daniel Soudry and Yair Carmon, No bad local minima: Data independent training error guarantees for multilayer neural networks , arXiv preprint arXiv:1605.08361 (2016)
Original
2016
Earlier work this paper cites.
Itay Safran and Ohad Shamir, On the quality of the initial basin in overspecified neural networks , International Conference on Machine Learning, 2016, pp. 774–782
2016
Earlier work this paper cites.
C Daniel Freeman and Joan Bruna, Topology and geometry of half-rectified network optimization , International Conference on Learning Representations (2017)
2017
Earlier work this paper cites.
Quynh Nguyen and Matthias Hein, The loss surface of deep and wide neural networks , Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, pp. 2603–2612
2017
Earlier work this paper cites.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli, Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice , Advances in neural information processing systems, 2017, pp. 4785–4795
2017
Earlier work this paper cites.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein, Deep information propagation , International Conference on Learning Representations (2017)
2017
Earlier work this paper cites.
Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via information , arXiv preprint arXiv:1703.00810 (2017)
Original
2017
Earlier work this paper cites.
Ge Yang and Samuel Schoenholz, Mean field residual networks: On the edge of chaos , Advances in neural information processing systems, 2017, pp. 7103–7114
2017
Earlier work this paper cites.
Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, and Inderjit S. Dhillon, Recovery guarantees for one-hidden-layer neural networks , Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, pp. 4140–4149
2017
Earlier work this paper cites.

Similar

Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang, Learning and generalization in overparameterized neural networks, going beyond two layers , arXiv preprint arXiv:1811.04918 (2018)
Original
2018
Cited alongside, same era.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song, A convergence theory for deep learning via over-parameterization , arXiv preprint arXiv:1811.03962 (2018)
Original
2018
Cited alongside, same era.
Lenaic Chizat and Francis Bach, A note on lazy training in supervised differentiable programming , arXiv preprint arXiv:1812.07956 (2018)
Original
2018
Cited alongside, same era.
Lénaïc Chizat and Francis Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport , Advances in Neural Information Processing Systems, 2018, pp. 3040–3050
2018
Cited alongside, same era.
Yaim Cooper, The loss landscape of overparameterized neural networks , arXiv preprint arXiv:1804.10200 (2018)
Original
2018
Cited alongside, same era.
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz, Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks , Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 873–882
2018
Cited alongside, same era.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani, Gaussian process behaviour in wide deep neural networks , International Conference on Learning Representations, 2018
2018
Cited alongside, same era.
Simon Du and Jason Lee, On the power of over-parametrization in neural networks with quadratic activation , Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 1329–1338
2018
Cited alongside, same era.
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai, Gradient descent finds global minima of deep neural networks , arXiv preprint arXiv:1811.03804 (2018)
Original
2018
Cited alongside, same era.
Mario Geiger, Stefano Spigler, Stéphane d’Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart, The jamming transition as a paradigm to understand the loss landscape of deep neural networks , arXiv preprint arXiv:1809.09349 (2018)
Original
2018
Cited alongside, same era.
Boris Hanin, Which neural net architectures give rise to exploding and vanishing gradients? , Advances in Neural Information Processing Systems 31, 2018, pp. 580–589
2018
Cited alongside, same era.
Nhat Ho, Tan Nguyen, Ankit Patel, Anima Anandkumar, Michael I Jordan, and Richard G Baraniuk, Neural rendering model: Joint generation and prediction for semi-supervised learning , arXiv preprint arXiv:1811.02657 (2018)
Original
2018
Cited alongside, same era.

Then

Quynh Nguyen and Matthias Hein, Optimization landscape and expressivity of deep cnns , International Conference on Machine Learning, 2018, pp. 3727–3736
2018
Later among the works it cites.
Grant M Rotskoff and Eric Vanden-Eijnden, Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error , arXiv preprint arXiv:1805.00915 (2018)
Original
2018
Later among the works it cites.
Stefano Spigler, Mario Geiger, Stéphane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart, A jamming transition from under-to over-parametrization affects loss landscape and generalization , arXiv preprint arXiv:1810.09665 (2018)
Original
2018
Later among the works it cites.
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee, Theoretical insights into the optimization landscape of over-parameterized shallow neural networks , IEEE Transactions on Information Theory (2018)
2018
Later among the works it cites.
Justin Sirignano and Konstantinos Spiliopoulos, Mean field analysis of neural networks , arXiv preprint arXiv:1805.01053 (2018)
Original
2018
Later among the works it cites.
Luca Venturi, Afonso Bandeira, and Joan Bruna, Spurious valleys in two-layer neural network optimization landscapes , arXiv preprint arXiv:1802.06384 (2018)
Original
2018
Later among the works it cites.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma, On the margin theory of feedforward neural networks , arXiv preprint arXiv:1810.05369 (2018)
Original
2018
Later among the works it cites.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington, Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks , Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 5393–5402
2018
Later among the works it cites.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu, Stochastic gradient descent optimizes over-parameterized deep relu networks , arXiv preprint arXiv:1811.08888 (2018)
Original
2018
Later among the works it cites.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh, Gradient descent provably optimizes over-parameterized neural networks , International Conference on Learning Representations, 2019
2019
Closest in time.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison, Deep convolutional networks as shallow gaussian processes , International Conference on Learning Representations, 2019
2019
Closest in time.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart, Scaling description of generalization with number of parameters in deep learning , arXiv preprint arXiv:1901.01608 (2019)
Original
2019
Closest in time.
Adel Javanmard, Marco Mondelli, and Andrea Montanari, Analysis of a two-layer neural network via displacement convexity , arXiv preprint arXiv:1901.01375 (2019)
Original
2019
Closest in time.
Ping Li and Phan-Minh Nguyen, On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training , International Conference on Learning Representations, 2019
2019
Closest in time.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein, Bayesian deep convolutional networks with many channels are gaussian processes , International Conference on Learning Representations, 2019
2019
Closest in time.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz, A mean field theory of batch normalization , International Conference on Learning Representations, 2019
2019
Closest in time.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie, Small nonlinearities in activation functions create bad local minima in neural networks , International Conference on Learning Representations, 2019
2019
Closest in time.
Stéphane Mallat, Understanding deep convolutional networks , Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374
2065
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…