2018

Deep Limits of Residual Neural Networks

Thorpe, Matthew, van Gennip, Yves

Understand

Neural networks have been very successful in many applications; we often, however, lack a theoretical understanding of what the neural networks are actually learning.

This problem emerges when trying to generalise to new data sets.
The contribution of this paper is to show that, for the residual neural network model, the deep layer limit coincides with a parameter estimation problem for a nonlinear ordinary differential equation.
In particular, whilst it is known that the residual neural network model is a discretisation of an ordinary differential equation, we show convergence in a variational sense.

Built on

On a stochastic approximation method
K.L. Chung · 1954
Earlier work this paper cites.
Analysis of recursive stochastic algorithms
L. Ljung · 1977
Earlier work this paper cites.
Elliptic problems in nonsmooth domains
P. Grisvard · 1985
Earlier work this paper cites.
The utility driven dynamic error propagation network
A. J. Robinson and F. Fallside · 1987
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
G. Cybenko · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
K. Hornik, M. Stinchcombe, and H. White · 1989
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
R. J. Williams and D. Zipser · 1989
Earlier work this paper cites.
Identification and control of dynamical systems using neural networks
K. S. Narendra and K. Parthasarathy · 1990
Earlier work this paper cites.
Introduction to the theory of neural computation
J. Hertz, A. Krogh, and R. G. Palmer · 1991
Earlier work this paper cites.
Untersuchungen zu dynamischen neuronalen netzen
S. Hochreiter · 1991
Earlier work this paper cites.
Approximation capabilities of multilayer feedforward networks
K. Hornik · 1991
Earlier work this paper cites.
Improving generalization performance using double backpropagation
H. Drucker and Y. Le Cun · 1992
Earlier work this paper cites.
Neural networks and complexity theory
P. Orponen · 1992
Earlier work this paper cites.
Introduction to artificial neural systems
J. M. Zurada · 1992
Earlier work this paper cites.
An Introduction to Γ \Gamma -Convergence
G. Dal Maso · 1993
Earlier work this paper cites.
Digital Neural Networks
S. Y. Kung · 1993
Earlier work this paper cites.
Finite impulse response neural networks for autoregressive time series prediction
E. A. Wan · 1993
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Y. Bengio, P. Simard, and P. Frasconi · 1994
Earlier work this paper cites.
Fundamentals of artificial neural networks
M. H. Hassoun · 1995
Earlier work this paper cites.
Neural networks in computational science and engineering
G. Cybenko · 1996
Earlier work this paper cites.
Neural networks: a comprehensive foundation
S. Haykin · 1999
Earlier work this paper cites.
Discrete mathematics of neural networks: selected topics
M. Anthony · 2001
Earlier work this paper cites.
Γ \Gamma -Convergence for Beginners
A. Braides · 2002
Earlier work this paper cites.
Sobolev spaces
R. A. Adams and J. J. F. Fournier · 2003
Earlier work this paper cites.
Lie groups, Lie algebras, and representations: an elementary introduction
B. C. Hall · 2003
Earlier work this paper cites.
Multidimensional real analysis I: differentiation
J. J. Duistermaat and J. A. C. Kolk · 2004
Earlier work this paper cites.
Feature selection, L1 vs. L2 regularization, and rotational invariance
A. Y. Ng · 2004
Earlier work this paper cites.
Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems
R. J. LeVeque · 2007
Earlier work this paper cites.
Sparse feature learning for deep belief networks
M. A. Ranzato, Y.-L. Boureau, and Y. LeCunn · 2008
Earlier work this paper cites.
Measuring invariances in deep networks
I. Goodfellow, H. Lee, Q. V. Le, A. Saxe, and A. Y. Ng · 2009
Earlier work this paper cites.
Ordinary Differential Equations
J. K. Hale · 2009
Earlier work this paper cites.
What is the best multi-stage architecture for object recognition?
K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun · 2009
Earlier work this paper cites.
A First Course in Sobolev Spaces
G. Leoni · 2009
Earlier work this paper cites.

Similar

Numerical analysis
R. L. Burden and J. D. Faires · 2010
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Cited alongside, same era.
Rectified linear units improve restricted boltzmann machines
V. Nair and G. E. Hinton · 2010
Cited alongside, same era.
Improving neural networks by preventing co-adaptation of feature detectors
Original
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov · 2012
Cited alongside, same era.
Neural network engineering in dynamic control systems
K. J. Hunt, G. R. Irwin, and K. Warwick · 2012
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Cited alongside, same era.
Neural networks and analog computation: beyond the Turing limit
H. T. Siegelmann · 2012
Cited alongside, same era.
Γ \Gamma -convergence of graph Ginzburg-Landau functionals
Y. van Gennip and A. L. Bertozzi · 2012
Cited alongside, same era.
Improving deep neural networks for LVCSR using rectified linear units and dropout
G. E. Dahl, T. N. Sainath, and G. E. Hinton · 2013
Cited alongside, same era.
Rectifier nonlinearities improve neural network acoustic models
A. L. Maas, A. Y. Hannun, and A. Y. Ng · 2013
Cited alongside, same era.
Regularization of neural networks using dropconnect
L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus · 2013
Cited alongside, same era.
On rectified linear units for speech processing
M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, and G. E. Hinton · 2013
Cited alongside, same era.

Then

Loss functions for image restoration with neural networks
H. Zhao, O. Gallo, I. Frosio, and J. Kautz · 2017
Later among the works it cites.
Reversible architectures for arbitrarily deep residual neural networks
B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham · 2018
Closest in time.
Deep relaxation: partial differential equations for optimizing deep neural networks
P. Chaudhari, A. Oberman, S. Osher, S. Soatto, and G. Carlier · 2018
Closest in time.
Neural ordinary differential equations
T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud · 2018
Closest in time.
A variational approach to the consistency of spectral clustering
N. García Trillos and D. Slepčev · 2018
Closest in time.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud · 2018
Closest in time.
Never look back - a modified EnKF method and its application to the training of neural networks without back propagation
Original
E. Haber, F. Lucka, and L. Ruthotto · 2018
Closest in time.
Learning across scales—multiscale methods for convolution neural networks
E. Haber, L. Ruthotto, E. Holtham, and S.-H. Jun · 2018
Closest in time.
Deep linear networks with arbitrary loss: All local minima are global
T. Laurent and J. Brecht · 2018
Closest in time.
Maximum principle based algorithms for deep learning
Q. Li, L. Chen, C. Tai, and W. E · 2018
Closest in time.
Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations
Y. Lu, A. Zhong, Q. Li, and B. Dong · 2018
Closest in time.
On the implicit bias of dropout
P. Mianjy, R. Arora, and R. Vidal · 2018
Closest in time.
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta · 2018
Closest in time.
Lipschitz regularized deep neural networks converge and generalize
Original
A. M. Oberman and J. Calder · 2018
Closest in time.
Deep neural networks motivated by partial differential equations
L. Ruthotto and E. Haber · 2018
Closest in time.
Low-cost parameterizations of deep convolution neural networks
Original
E. Treister, L. Ruthotto, M. Sharoni, S. Zafrani, and E. Haber · 2018
Closest in time.
A mathematical theory of deep convolutional neural networks for feature extraction
T. Wiatowski and H. Bölcskei · 2018
Closest in time.
Large data and zero noise limits of graph-based semi-supervised learning algorithms
M. M. Dunlop, D. Slepčev, A. M. Stuart, and M. Thorpe · 2019
Closest in time.
A mean-field optimal control formulation of deep learning
W. E, J. Han, and Q. Li · 2019
Closest in time.
A mean-field optimal control formulation of deep learning
W. E, J. Han, and Q. Li · 2019
Closest in time.
Deep learning: An introduction for applied mathematicians
C. F. Higham and D. J. Higham · 2019
Closest in time.
Ensemble Kalman inversion: A derivative-free technique for machine learning tasks
N. B. Kovachki and A. M. Stuart · 2019
Closest in time.
Partial differential equation regularization for supervised machine learning
Original
A. M. Oberman · 2019
Closest in time.
Analysis of p p -Laplacian regularization in semi-supervised learning
D. Slepčev and M. Thorpe · 2019
Closest in time.
Blended coarse gradient descent for full quantization of deep neural networks
P. Yin, S. Zhang, J. Lyu, S. Osher, Y. Qi, and J. Xin · 2019
Closest in time.
Forward stability of ResNet and its variants
L. Zhang and H. Schaeffer · 2019
Closest in time.
Deep residual learning via large sample mean-field optimization
Original
L. Bo, A. Capponi, and H. Liao · 2020
Closest in time.
On the consistency of graph-based Bayesian semi-supervised learning of the scalability of sampling algorithms
N. García Trillos, Z. Kaplan, T. Samakhoana, and D. Sanz-Alonso · 2020
Closest in time.
A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth
Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying · 2020
Closest in time.
PDE-based group equivariant convolutional neural networks
Original
B. M. N. Smets, J. Portegies, E. J. Bekkers, and R. Duits · 2020
Closest in time.
Norm-preservation: Why residual networks can become extremely deep?
A. Zaeemzadeh, N. Rahnavard, and M. Shah · 2020
Closest in time.
Structure-preserving deep learning
E. Celledoni, M. J. Ehrhardt, C. Etmann, R. I. McLachlan, B. Owren, C.-B. Schönlieb, and F. Sherry · 2021
Closest in time.
Scaling properties of deep residual networks
A.-S. Cohen, R. Cont, A. Rossier, and R. Xu · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…