2022

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

Li, Mufan Bill, Nica, Mihai, Roy, Daniel M.

Understand

The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer.

In this work, we study the distribution of this random matrix.
Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate.
However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-width analyses ignore the microscopic fluctuations from layer to layer, but these fluctuations accumulate over many layers.

Built on

“Wide neural networks of any depth evolve as linear models under gradient descent”, 2019
Original
Jaehoon Lee et al · 1902
Earlier work this paper cites.
“Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation”, 2019
Original
Greg Yang · 1902
Earlier work this paper cites.
“Bayesian learning for neural networks”
Radford Neal · 1995
Earlier work this paper cites.
“Multidimensional diffusion processes”
Daniel Stroock and SR Varadhan · 1997
Earlier work this paper cites.
“Tensor programs ii: Neural tangent kernel for any architecture”, 2020
Original
Greg Yang · 2006
Earlier work this paper cites.
“Kernel methods for deep learning”
Youngmin Cho and Lawrence Saul · 2009
Earlier work this paper cites.
“Markov processes: characterization and convergence”
Stewart Ethier and Thomas Kurtz · 2009
Earlier work this paper cites.
“Feature Learning in Infinite-Width Neural Networks”
Original
Greg Yang and Edward. Hu · 2011
Earlier work this paper cites.
“Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?”, 2020
Original
Mariia Seleznova and Gitta Kutyniok · 2012
Earlier work this paper cites.
“Brownian motion and stochastic calculus”
Ioannis Karatzas and Steven Shreve · 2012
Earlier work this paper cites.
“Delving deep into rectifiers: Surpassing human-level performance on imagenet classification”
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun · 2015
Earlier work this paper cites.
“Deep information propagation”
Original
Samuel Schoenholz, Justin Gilmer, Surya Ganguli and Jascha Sohl-Dickstein · 2016
Earlier work this paper cites.
“Deep residual learning for image recognition”
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun · 2016
Earlier work this paper cites.
“Mean field residual networks: on the edge of chaos”
Greg Yang and Samuel Schoenholz · 2017
Earlier work this paper cites.
“SymPy: symbolic computing in Python”
Aaron Meurer et al · 2017
Earlier work this paper cites.
“Deep Neural Networks as Gaussian Processes”
Jaehoon Lee et al · 2018
Earlier work this paper cites.
“Neural tangent kernel: Convergence and generalization in neural networks”
Original
Arthur Jacot, Franck Gabriel and Clément Hongler · 2018
Earlier work this paper cites.
“Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach”, 2018
Original
Grant. Rotskoff and Eric Vanden-Eijnden · 2018
Earlier work this paper cites.

Similar

“On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport”, 2018
Original
Lenaic Chizat and Francis Bach · 2018
Cited alongside, same era.
“Mean Field Analysis of Neural Networks: A Law of Large Numbers”, 2018
Original
Justin Sirignano and Konstantinos Spiliopoulos · 2018
Cited alongside, same era.
“A mean field view of the landscape of two-layer neural networks”
Song Mei, Andrea Montanari and Phan-Minh Nguyen · 2018
Cited alongside, same era.
“How to Start Training: The Effect of Initialization and Architecture”
Boris Hanin and David Rolnick · 2018
Cited alongside, same era.
“Bayesian deep convolutional networks with many channels are gaussian processes”
Original
Roman Novak et al · 2018
Cited alongside, same era.
“Gradient descent finds global minima of deep neural networks”
Simon Du et al · 2019
Cited alongside, same era.
“A convergence theory for deep learning via over-parameterization”
Zeyuan Allen-Zhu, Yuanzhi Li and Zhao Song · 2019
Cited alongside, same era.
“On Lazy Training in Differentiable Programming”
Lénaı̈c Chizat, Edouard Oyallon and Francis Bach · 2019
Cited alongside, same era.
“On exact computation with an infinitely wide neural net”
Sanjeev Arora et al · 2019
Cited alongside, same era.
“Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks”
Original
Ziwei Ji and Matus Telgarsky · 2019
Cited alongside, same era.
“Generalization of two-layer neural networks: An asymptotic viewpoint”
Jimmy Ba et al · 2019
Cited alongside, same era.
“Finite Depth and Width Corrections to the Neural Tangent Kernel”
Boris Hanin and Mihai Nica · 2019
Cited alongside, same era.

Then

“How Much Over-parameterization Is Sufficient to Learn Deep Re{LU} Networks?”
Zixiang Chen, Yuan Cao, Difan Zou and Quanquan Gu · 2021
Later among the works it cites.
“Deep learning: a statistical viewpoint”
Peter Bartlett, Andrea Montanari and Alexander Rakhlin · 2021
Later among the works it cites.
“On the Random Conjugate Kernel and Neural Tangent Kernel”
Zhengmian Hu and Heng Huang · 2021
Later among the works it cites.
“The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization”
Mufan Li, Mihai Nica and Dan Roy · 2021
Later among the works it cites.
“Exact marginal prior distributions of finite Bayesian neural networks”
Jacob Zavatone-Veth and Cengiz Pehlevan · 2021
Later among the works it cites.
“Precise characterization of the prior predictive distribution of deep ReLU networks”
Lorenzo Noci et al · 2021
Later among the works it cites.
“Asymptotics of representation learning in finite Bayesian neural networks”
Jacob Zavatone-Veth, Abdulkadir Canatar, Ben Ruben and Cengiz Pehlevan · 2021
Later among the works it cites.
“Deep Networks and the Multiple Manifold Problem”
Sam Buchanan, Dar Gilboa and John Wright · 2021
Later among the works it cites.
“Stable ResNet”
Soufiane Hayou et al · 2021
Later among the works it cites.
“Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping”
Original
James Martens et al · 2021
Later among the works it cites.
“Foundations of Modern Probability”, Probability theory and stochastic modelling
O. Kallenberg · 2021
Later among the works it cites.
“Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”
Original
Greg Yang et al · 2022
Closest in time.
“High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation”
Original
Jimmy Ba et al · 2022
Closest in time.
“The principles of deep learning theory”
Daniel Roberts, Sho Yaida and Boris Hanin · 2022
Closest in time.
“Correlation Functions in Random Fully Connected Neural Networks at Finite Width”
Original
Boris Hanin · 2022
Closest in time.
“Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers”
Original
Guodong Zhang, Aleksandar Botev and James Martens · 2022
Closest in time.
“Activation function design for deep networks: linearity and effective initialisation”
Michael Murray, Vinayak Abrol and Jared Tanner · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…