2019

Asymptotics of Wide Networks from Feynman Diagrams

Dyer, Ethan, Gur-Ari, Guy

Understand

Understanding the asymptotic behavior of wide networks is of considerable interest.

In this work, we present a general method for analyzing this large width behavior.
The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals.
We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent.

Built on

Space - time approach to quantum electrodynamics
R. P. Feynman · 1949
Earlier work this paper cites.
A Planar Diagram Theory for Strong Interactions
Gerard ’t Hooft · 1973
Earlier work this paper cites.
Priors for Infinite Networks
Radford M. Neal · 1996
Earlier work this paper cites.
Numerical Linear Algebra
Lloyd N. Trefethen and David Bau · 1997
Earlier work this paper cites.
Kernel methods for deep learning
Youngmin Cho and Lawrence K. Saul · 2009
Earlier work this paper cites.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Earlier work this paper cites.
Eigenvalues of the hessian in deep learning: Singularity and beyond
Original
Levent Sagun, Leon Bottou, and Yann LeCun · 2016
Earlier work this paper cites.
Sgd learns the conjugate kernel class of the network
Amit Daniely · 2017
Earlier work this paper cites.
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Original
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli · 2017
Earlier work this paper cites.
Geometry of neural network loss surfaces via random matrix theory
Jeffrey Pennington and Yasaman Bahri · 2017
Earlier work this paper cites.
A correspondence between random neural networks and statistical field theory
Original
Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2017
Earlier work this paper cites.
Nonlinear random matrix theory for deep learning
Jeffrey Pennington and Pratik Worah · 2017
Earlier work this paper cites.
Empirical analysis of the hessian of over-parametrized neural networks
Original
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou · 2017
Earlier work this paper cites.

Similar

Deep neural networks as gaussian processes
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri · 2018
Cited alongside, same era.
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Original
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Cited alongside, same era.
Gaussian process behaviour in wide deep neural networks
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani · 2018
Cited alongside, same era.
The spectrum of the fisher information matrix of a single-hidden-layer neural network
Jeffrey Pennington and Pratik Worah · 2018
Cited alongside, same era.
Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks
Original
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington · 2018
Cited alongside, same era.
Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks
Original
Minmin Chen, Jeffrey Pennington, and Samuel S Schoenholz · 2018
Cited alongside, same era.
Gradient descent provably optimizes over-parameterized neural networks
Original
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh · 2018
Cited alongside, same era.
Gradient descent finds global minima of deep neural networks
Original
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai · 2018
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Cited alongside, same era.
On Lazy Training in Differentiable Programming
Original
Lenaic Chizat, Edouard Oyallon, and Francis Bach · 2018
Cited alongside, same era.
Products of many large random matrices and gradients in deep neural networks
Original
Boris Hanin and Mihai Nica · 2018
Cited alongside, same era.
Gradient descent happens in a tiny subspace
Original
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer · 2018
Cited alongside, same era.

Then

Bayesian deep convolutional networks with many channels are gaussian processes
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein · 2019
Closest in time.
Deep convolutional networks as shallow gaussian processes
Adria Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison · 2019
Closest in time.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation
Original
Greg Yang · 2019
Closest in time.
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Original
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Closest in time.
Scaling description of generalization with number of parameters in deep learning
Original
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart · 2019
Closest in time.
Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
Original
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang · 2019
Closest in time.
Dynamics of deep neural networks and neural tangent hierarchy, 2019
Jiaoyang Huang and Horng-Tzer Yau · 2019
Closest in time.
Asymptotics of wide networks from feynman diagrams
Ethan Dyer and Guy Gur-Ari · 2019
Closest in time.
Limitations of Lazy Training of Two-layers Neural Networks
Original
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari · 2019
Closest in time.
Disentangling feature and lazy learning in deep neural networks: an empirical study
Original
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart · 2019
Closest in time.
An investigation into neural net optimization via hessian eigenvalue density
Original
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao · 2019
Closest in time.
Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians
Original
Vardan Papyan · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…