2017

Mathematics of Deep Learning

Vidal, Rene, Bruna, Joan, Giryes, Raja et al.

Understand

Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification.

However, the mathematical reasons for this success remain elusive.
This tutorial will review recent work that aims to provide a mathematical justification for several properties of deep networks, such as global optimality, geometric stability, and invariance of the learned representations.

Built on

Beyond regression: New tools for predictions and analysis in the behavioral science. Cambridge, MA, itd
PJ Werbos · 1974
Earlier work this paper cites.
Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams · 1988
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
G. Cybenko · 1989
Earlier work this paper cites.
Multilayer feedforward networks are universal approximators
K. Hornik, M. Stinchcombe, and H. White · 1989
Earlier work this paper cites.
Neural networks and principal component analysis: Learning from examples without local minima
P. Baldi and K. Hornik · 1989
Earlier work this paper cites.
Back propagation fails to separate where perceptrons succeed
Martin Brady, Raghu Raghavan, and Joseph Slawny · 1989
Earlier work this paper cites.
Approximation capabilities of multilayer feedforward networks
Kurt Hornik · 1991
Earlier work this paper cites.
Backpropagation converges for multi-layered networks and linearly-separable patterns
M. Gori and A. Tesi · 1991
Earlier work this paper cites.
On the problem of local minima in backpropagation
M. Gori and A. Tesi · 1992
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
G. Hinton and D. Van Camp · 1993
Earlier work this paper cites.
Approximation and estimation bounds for artificial neural networks
Andrew R Barron · 1994
Earlier work this paper cites.
Numerical Optimization
Stephen J Wright and Jorge Nocedal · 1999
Earlier work this paper cites.
An overview of statistical learning theory
Vladimir N Vapnik · 1999
Earlier work this paper cites.
The information bottleneck method
N. Tishby, F. C. Pereira, and W. Bialek · 2000
Earlier work this paper cites.
Boosting algorithms as gradient descent
Llew Mason, Jonathan Baxter, Peter L. Bartlett, and Marcus R. Frean · 2000
Earlier work this paper cites.
Greedy function approximation: a gradient boosting machine
Jerome H Friedman · 2001
Earlier work this paper cites.
Rademacher and Gaussian complexities: risk bounds and structural results
Peter L Bartlett and Shahar Mendelson · 2002
Earlier work this paper cites.
Vapnik-Chervonenkis dimension of neural nets
P. Bartlett and W. Maass · 2003
Earlier work this paper cites.
Learning over sets using kernel principal angles
L. Wolf and A. Shashua · 2003
Earlier work this paper cites.
Convex neural networks
Yoshua Bengio, Nicolas L Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte · 2005
Earlier work this paper cites.
Near-optimal signal recovery from random projections: Universal encoding strategies?
E. J. Candès and T. Tao · 2006
Earlier work this paper cites.
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, R Socher, Li-Jia Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
On the set of images modulo viewpoint and contrast changes
G. Sundaramoorthi, P. Petersen, V. S. Varadarajan, and S. Soatto · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
V. Nair and G.E. Hinton · 2010
Earlier work this paper cites.
Online learning for matrix factorization and sparse coding
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro · 2010
Earlier work this paper cites.
Object detection with discriminatively trained part-based models
P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan · 2010
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.

Similar

Group invariant scattering
Stéphane Mallat · 2012
Cited alongside, same era.
The convex geometry of linear inverse problems
V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky · 2012
Cited alongside, same era.
Robustness and generalization
Huan Xu and Shie Mannor · 2012
Cited alongside, same era.
Invariant scattering convolution networks
J. Bruna and S. Mallat · 2013
Cited alongside, same era.
Minimizing finite sums with the stochastic average gradient
Mark Schmidt, Nicolas Le Roux, and Francis Bach · 2013
Cited alongside, same era.
A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion
Yangyang Xu and Wotao Yin · 2013
Cited alongside, same era.
Improving deep neural networks for lvcsr using rectified linear units and dropout
George E Dahl, Tara N Sainath, and Geoffrey E Hinton · 2013
Cited alongside, same era.
Rectifier nonlinearities improve neural network acoustic models
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng · 2013
Cited alongside, same era.
On rectified linear units for speech processing
Matthew D. Zeiler, M. Ranzato, Rajat Monga, M. Mao, K. Yang, Quoc Viet Le, Patrick Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean, and Geoffry E. Hinton · 2013
Cited alongside, same era.
Sparse subspace clustering: Algorithm, theory, and applications
E. Elhamifar and R. Vidal · 2013
Cited alongside, same era.
Auto-encoding variational bayes
Original
D. Kingma and M. Welling · 2013
Cited alongside, same era.
Dropout: A simple way to prevent neural networks from overfitting
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov · 2014
Cited alongside, same era.

Then

Discriminative robust transformation learning
J. Huang, Q. Qiu, G. Sapiro, and R. Calderbank · 2015
Later among the works it cites.
Variational dropout and the local reparameterization trick
D. Kingma, T. Salimans, and M. Welling · 2015
Later among the works it cites.
Identity mappings in deep residual networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Deep neural networks with random gaussian weights: A universal classification strategy?
R. Giryes, G. Sapiro, and A. Bronstein · 2016
Later among the works it cites.
Visual representations: Defining properties and deep approximations
S. Soatto and A. Chiuso · 2016
Later among the works it cites.
Information dropout: Learning optimal representations through noisy computation
Original
A. Achille and S. Soatto · 2016
Later among the works it cites.
Deep variational information bottleneck
Original
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy · 2016
Later among the works it cites.
Local entropy as a measure for sampling solutions in constraint satisfaction problems
C. Baldassi, A. Ingrosso, C. Lucibello, L. Saglietti, and R. Zecchina · 2016
Later among the works it cites.
On the quality of the initial basin in overspecified neural networks
Itay Safran and Ohad Shamir · 2016
Later among the works it cites.
Understanding deep convolutional networks
Stéphane Mallat · 2016
Later among the works it cites.
Geometric deep learning: going beyond Euclidean data
Original
M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst · 2016
Later among the works it cites.
Deep neural networks with random Gaussian weights: a universal classification strategy?
R. Giryes, G. Sapiro, and A. Bronstein · 2016
Later among the works it cites.
Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
Original
V. Papyan, Y. Romano, and M. Elad · 2016
Later among the works it cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Later among the works it cites.
Train faster, generalize better: Stability of stochastic gradient descent
Moritz Hardt, Benjamin Recht, and Yoram Singer · 2016
Later among the works it cites.
Opening the black box of deep neural networks via information
Original
Ravid Shwartz-Ziv and Naftali Tishby · 2017
Closest in time.
Sharp minima can generalize for deep nets
Original
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio · 2017
Closest in time.
Entropy-SGD: Biasing gradient descent into wide valleys
P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina · 2017
Closest in time.
Deep relaxation: partial differential equations for optimizing deep neural networks
Original
Pratik Chaudhari, Adam Oberman, Stanley Osher, Stefano Soatto, and Guillaume Carlier · 2017
Closest in time.
Topology and geometry of half-rectified network optimization
C Daniel Freeman and Joan Bruna · 2017
Closest in time.
Robust large margin deep neural networks
J. Sokolić, R. Giryes, G. Sapiro, and M. Rodrigues · 2017
Closest in time.
Generalization error of invariant classifiers
J. Sokolić, R. Giryes, G. Sapiro, and M. Rodrigues · 2017
Closest in time.
Global optimality in neural network training
B. Haeffele and R. Vidal · 2017
Closest in time.
Tradeoffs between convergence speed and reconstruction accuracy in inverse problems
Original
R. Giryes, Y. Eldar, A. Bronstein, and G. Sapiro · 2017
Closest in time.
Understanding deep learning requires rethinking generalization
C. Zhang, S. Bengio, M. Hardt, and B. Recht · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…