2018

Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Karakida, Ryo, Akaho, Shotaro, Amari, Shun-ichi

Understand

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs).

The present study reveals novel statistics of FIM that are universal among a wide class of DNNs.
To this end, we use random weights and large width limits, which enables us to utilize mean field theories.
We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value.

Built on

A method of statistical neurodynamics
Shun-ichi Amari · 1974
Earlier work this paper cites.
Exact solution for on-line learning in multilayer neural networks
David Saad and Sara A Solla · 1995
Earlier work this paper cites.
A regularity condition of the information matrix of a multilayer perceptron network
Kenji Fukumizu · 1996
Earlier work this paper cites.
Flat minima
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Efficient backprop
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller · 1998
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-ichi Amari · 1998
Earlier work this paper cites.
Online learning and stochastic approximations
Léon Bottou · 1998
Earlier work this paper cites.
Adaptive method of realizing natural gradient learning for multilayer perceptrons
Shun-Ichi Amari, Hyeyoung Park, and Kenji Fukumizu · 2000
Earlier work this paper cites.
Adaptive natural gradient learning algorithms for various stochastic models
Hyeyoung Park, Shun-ichi Amari, and Kenji Fukumizu · 2000
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
What does classifying more than 10,000 image categories tell us?
Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei · 2010
Earlier work this paper cites.
Revisiting natural gradient for deep networks
Original
Razvan Pascanu and Yoshua Bengio · 2013
Earlier work this paper cites.
On the number of linear regions of deep neural networks
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio · 2014
Earlier work this paper cites.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Earlier work this paper cites.

Similar

Riemannian metrics for neural networks I: feedforward networks
Yann Ollivier · 2015
Cited alongside, same era.
Optimizing neural networks with kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Cited alongside, same era.
Norm-based capacity control in neural networks
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Exponential expressivity in deep neural networks through transient chaos
Ben Poole, Subhaneil Lahiri, Maithreyi Raghu, Jascha Sohl-Dickstein, and Surya Ganguli · 2016
Cited alongside, same era.
Deep information propagation
Original
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein · 2016
Cited alongside, same era.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Cited alongside, same era.
Optimal architectures in a solvable model of deep networks
Jonathan Kadmon and Haim Sompolinsky · 2016
Cited alongside, same era.
The landscape of empirical risk for non-convex losses
Original
Song Mei, Yu Bai, and Andrea Montanari · 2016
Cited alongside, same era.
On large-batch training for deep learning: Generalization gap and sharp minima
Original
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang · 2016
Cited alongside, same era.
On the quality of the initial basin in overspecified neural networks
Itay Safran and Ohad Shamir · 2016
Cited alongside, same era.
Geometry of neural network loss surfaces via random matrix theory
Jeffrey Pennington and Yasaman Bahri · 2017
Cited alongside, same era.

Then

Exponentially vanishing sub-optimal local minima in multilayer neural networks
Original
Daniel Soudry and Elad Hoffer · 2017
Later among the works it cites.
The loss surface of deep and wide neural networks
Quynh Nguyen and Matthias Hein · 2017
Later among the works it cites.
An analytical formula of population gradient for two-layered ReLU network and its applications in convergence and critical point analysis
Yuandong Tian · 2017
Later among the works it cites.
Fisher-Rao metric, geometry, and complexity of neural networks
Original
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes · 2017
Later among the works it cites.
Empirical analysis of the hessian of over-parametrized neural networks
Original
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou · 2017
Later among the works it cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017
Later among the works it cites.
Deep neural networks as gaussian processes
Original
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2017
Later among the works it cites.
Spectrally-normalized margin bounds for neural networks
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky · 2017
Later among the works it cites.
Sharp minima can generalize for deep nets
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio · 2017
Later among the works it cites.
Understanding generalization and stochastic gradient descent
Original
Samuel L Smith and Quoc V Le · 2017
Later among the works it cites.
The emergence of spectral universality in deep networks
Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli · 2018
Closest in time.
Exploring the function space of deep-learning machines
Bo Li and David Saad · 2018
Closest in time.
Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington · 2018
Closest in time.
Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks
Minmin Chen, Jeffrey Pennington, and Samuel S Schoenholz · 2018
Closest in time.
Towards understanding the role of over-parametrization in generalization of neural networks
Original
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…