2020

On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime

Wojtowytsch, Stephan

Understand

We describe a necessary and sufficient condition for the convergence to minimum Bayes risk when training two-layer ReLU-networks by gradient descent in the mean field regime with omni-directional initial parameter distribution.

This article extends recent results of Chizat and Bach to ReLU-activated networks and to the situation in which there are no parameters which exactly achieve MBR.
The condition does not depend on the initalization of parameters and concerns only the weak convergence of the realization of the neural network, not its parameter distribution.

Built on

A function not constant on a connected set of critical points
H. Whitney · 1935
Earlier work this paper cites.
The behavior of a function on its critical set
A. P. Morse · 1939
Earlier work this paper cites.
The measure of the critical values of differentiable maps
A. Sard · 1942
Earlier work this paper cites.
An approximate Morse-Sard theorem
J. Eells and J. McAlpin · 1968
Earlier work this paper cites.
Geometric Measure Theory
H. Federer · 1969
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
G. Cybenko · 1989
Earlier work this paper cites.
Approximation capabilities of multilayer feedforward networks
K. Hornik · 1991
Earlier work this paper cites.
Universal approximation bounds for superpositions of a sigmoidal function
A. R. Barron · 1993
Earlier work this paper cites.
Toward a precise smoothness hypothesis in Sard’s theorem
S. M. Bates · 1993
Earlier work this paper cites.
Multilayer feedforward networks with a nonpolynomial activation function can approximate any function
M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken · 1993
Earlier work this paper cites.
Infinite Dimensional Analysis: A Hitchhiker’s Guide
C. D. Aliprantis and K. Border · 1994
Earlier work this paper cites.
A convexity principle for interacting gases
R. J. McCann · 1997
Earlier work this paper cites.
The variational formulation of the Fokker–Planck equation
R. Jordan, D. Kinderlehrer, and F. Otto · 1998
Earlier work this paper cites.
The Morse-Sard theorem in Sobolev spaces
L. de Pascale · 2001
Earlier work this paper cites.
The geometry of dissipative evolution equations: the porous medium equation
F. Otto · 2001
Earlier work this paper cites.
Whitney’s example by way of Assouad’s embedding
P. Hajłasz · 2003
Earlier work this paper cites.
A Morse-Sard theorem for the distance function on Riemannian manifolds
L. Rifford · 2004
Earlier work this paper cites.
Gamma-convergence of gradient flows with applications to Ginzburg-Landau
E. Sandier and S. Serfaty · 2004
Earlier work this paper cites.
A nonsmooth Morse-Sard theorem for subanalytic functions
J. Bolte, A. Daniilidis, and A. Lewis · 2006
Earlier work this paper cites.
Morse-sard theorem for dc functions and mappings on ℝ 2 \mathbb{R}^{2}
D. Pavlica and L. Zajíček · 2006
Earlier work this paper cites.
Modern Methods in the Calculus of Variations: L p {L}^{p} Spaces
I. Fonseca and G. Leoni · 2007
Earlier work this paper cites.
Gradient flows: in metric spaces and in the space of probability measures
L. Ambrosio, N. Gigli, and G. Savaré · 2008
Earlier work this paper cites.
Transport equation and Cauchy problem for non-smooth vector fields
L. Ambrosio · 2008
Earlier work this paper cites.
A simple proof of the Morse-Sard theorem in Sobolev spaces
A. Figalli · 2008
Earlier work this paper cites.

Similar

Optimal transport: old and new
C. Villani · 2008
Cited alongside, same era.
Angewandte Funktionalanalysis: Funktionalanalysis, Sobolev-Räume und elliptische Differentialgleichungen
M. Dobrowolski · 2010
Cited alongside, same era.
Gamma-convergence of gradient flows on Hilbert and metric spaces and applications
S. Serfaty · 2011
Cited alongside, same era.
Emergence of rate-independent dissipation from viscous systems with wiggly energies
A. Mielke · 2012
Cited alongside, same era.
On the Morse-Sard property and level sets of Sobolev and BV functions
J. Bourgain, M. V. Korobkov, and J. Kristensen · 2013
Cited alongside, same era.
Variational analysis in Sobolev and BV spaces: applications to PDEs and optimization
H. Attouch, G. Buttazzo, and G. Michaille · 2014
Cited alongside, same era.
Understanding machine learning: From theory to algorithms
S. Shalev-Shwartz and S. Ben-David · 2014
Cited alongside, same era.
On the Morse–Sard property and level sets of W n , 1 {W}^{n,1} -Sobolev functions on ℝ n \mathbb{R}^{n}
J. Bourgain, M. V. Korobkov, and J. Kristensen · 2015
Cited alongside, same era.
Measure theory and fine properties of functions
L. C. Evans and R. F. Gariepy · 2015
Cited alongside, same era.
Dynamics of stochastic gradient algorithms
Original
Q. Li, C. Tai, and W. E · 2015
Cited alongside, same era.
Optimal transport for applied mathematicians
F. Santambrogio · 2015
Cited alongside, same era.
Sard theorems for Lipschitz functions and applications in optimization
L. Barbet, M. Dambrine, A. Daniilidis, and L. Rifford · 2016
Cited alongside, same era.

Then

High-dimensional probability: An introduction with applications in data science
R. Vershynin · 2018
Later among the works it cites.
On exact computation with an infinitely wide neural net
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang · 2019
Later among the works it cites.
Maximum mean discrepancy gradient flow
M. Arbel, A. Korba, A. Salim, and A. Gretton · 2019
Later among the works it cites.
A mean-field limit for certain deep neural networks
Original
D. Araújo, R. I. Oliveira, and D. Yukimura · 2019
Later among the works it cites.
On the inductive bias of neural tangent kernels
A. Bietti and J. Mairal · 2019
Later among the works it cites.
The effect of forest dislocations on the evolution of a phase-field model for plastic slip
P. W. Dondl, M. W. Kurzke, and S. Wojtowytsch · 2019
Later among the works it cites.
A priori estimates of the population risk for residual networks
Original
W. E, C. Ma, and Q. Wang · 2019
Later among the works it cites.
Barron spaces and the compositional function spaces for neural network models
Original
W. E, C. Ma, and L. Wu · 2019
Later among the works it cites.
Machine learning from a continuous viewpoint
Original
W. E, C. Ma, and L. Wu · 2019
Later among the works it cites.
A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
Original
W. E, C. Ma, and L. Wu · 2019
Later among the works it cites.
Analysis of the gradient descent algorithm for a deep neural network model with skip-connections
Original
W. E, C. Ma, Q. Wang, and L. Wu · 2019
Later among the works it cites.
Deep learning: An introduction for applied mathematicians
C. F. Higham and D. J. Higham · 2019
Later among the works it cites.
On the diffusion approximation of nonconvex stochastic gradient descent
W. Hu, C. J. Li, L. Li, and J.-G. Liu · 2019
Later among the works it cites.
Mean-field Langevin dynamics and energy landscape of neural networks
Original
K. Hu, Z. Ren, D. Siska, and L. Szpruch · 2019
Later among the works it cites.
A high-bias, low-variance introduction to machine learning for physicists
P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab · 2019
Later among the works it cites.
Mean field limit of the learning dynamics of multilayer neural networks
Original
P.-M. Nguyen · 2019
Later among the works it cites.
Mean field analysis of deep neural networks
Original
J. Sirignano and K. Spiliopoulos · 2019
Later among the works it cites.
Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss
Original
L. Chizat and F. Bach · 2020
Closest in time.
Barron functions and their representation
W. E and S. Wojtowytsch · 2020
Closest in time.
Kolmogorov width decay and poor approximators in machine learning: Shallow neural networks, random feature models and neural tangent kernels
Original
W. E and S. Wojtowytsch · 2020
Closest in time.
A rigorous framework for the mean field limit of multilayer neural networks
Original
P.-M. Nguyen and H. T. Pham · 2020
Closest in time.
Mean field analysis of neural networks: A law of large numbers
J. Sirignano and K. Spiliopoulos · 2020
Closest in time.
Can shallow neural networks beat the curse of dimensionality? A mean field training perspective
Original
S. Wojtowytsch and W. E · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…