2020

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

Chen, Zixiang, Cao, Yuan, Gu, Quanquan et al.

Understand

A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tangent kernel} (NTK).

However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the network weights to stay very close to their initialization throughout training, and cannot handle regularizers or gradient noises.
In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior.
This implies that the training loss converges linearly up to a certain accuracy.

Built on

Over parameterized two-level neural networks can learn near optimal feature representations
Original
Fang, C · 1910
Earlier work this paper cites.
Convex formulation of overparameterized deep neural networks
Original
Fang, C · 1911
Earlier work this paper cites.
Towards understanding the spectral bias of deep learning
Original
Cao, Y · 1912
Earlier work this paper cites.
Asymptotic evaluation of certain markov process expectations for large time. iv
Donsker, M. D · 1983
Earlier work this paper cites.
Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality
Otto, F · 2000
Earlier work this paper cites.
Rademacher and Gaussian complexities: Risk bounds and structural results
Bartlett, P. L · 2002
Earlier work this paper cites.
A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled mckean-vlasov dynamics
Original
Tzen, B · 2002
Earlier work this paper cites.
Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning
Original
Liu, C · 2003
Earlier work this paper cites.
Generalization error bounds for bayesian mixture algorithms
Meir, R · 2003
Earlier work this paper cites.
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups
Hinton, G · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A · 2012
Earlier work this paper cites.

Similar

Analysis and geometry of Markov diffusion operators
Bakry, D · 2013
Cited alongside, same era.
Understanding machine learning: From theory to algorithms
Shalev-Shwartz, S · 2014
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
Silver, D · 2016
Cited alongside, same era.
Breaking the curse of dimensionality with convex neural networks
Bach, F · 2017
Cited alongside, same era.
Spectrally-normalized margin bounds for neural networks
Bartlett, P. L · 2017
Cited alongside, same era.
On the global convergence of gradient descent for over-parameterized models using optimal transport
Chizat, L · 2018
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A · 2018
Cited alongside, same era.
On tighter generalization bound for deep neural networks: CNNs, ResNets, and beyond
Original
Li, X · 2018
Cited alongside, same era.
Learning overparameterized neural networks via stochastic gradient descent on structured data
Li, Y · 2018
Cited alongside, same era.
A mean field view of the landscape of two-layer neural networks
Mei, S · 2018
Cited alongside, same era.
Foundations of machine learning
Mohri, M · 2018
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Allen-Zhu, Z
Cited in the paper.

Then

Global convergence of langevin dynamics based algorithms for nonconvex optimization
Xu, P · 2018
Later among the works it cites.
Generalization bounds of stochastic gradient descent for wide and deep neural networks
Cao, Y · 2019
Later among the works it cites.
On lazy training in differentiable programming
Chizat, L · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Lee, J · 2019
Later among the works it cites.
Towards explaining the regularization effect of initial large learning rate in training neural networks
Li, Y · 2019
Later among the works it cites.
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
Mei, S · 2019
Later among the works it cites.
On learning over-parameterized neural networks: A functional approximation prospective
Su, L · 2019
Later among the works it cites.
Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel
Wei, C · 2019
Later among the works it cites.
Gradient descent optimizes over-parameterized deep ReLU networks
Zou, D · 2019
Later among the works it cites.
An improved analysis of training over-parameterized deep neural networks
Zou, D · 2019
Later among the works it cites.
Generalization error bounds of gradient descent for learning over-parameterized deep relu networks
Cao, Y · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…