2019

Pathological spectra of the Fisher information metric and its variants in deep neural networks

Karakida, Ryo, Akaho, Shotaro, Amari, Shun-ichi

Understand

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions.

Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth and sample size when the network has random weights and is sufficiently wide.
This study covers two widely-used FIMs for regression with linear output and for classification with softmax output.
Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending the width or sample size while the others are much smaller.

Built on

The normalization method for alleviating pathological sharpness in wide neural networks
Original
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari · 1906
Earlier work this paper cites.
A method of statistical neurodynamics
Shun-ichi Amari · 1974
Earlier work this paper cites.
Eigenvalues of covariance matrices: Application to neural-network learning
Yann LeCun, Ido Kanter, and Sara A Solla · 1991
Earlier work this paper cites.
A regularity condition of the information matrix of a multilayer perceptron network
Kenji Fukumizu · 1996
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-ichi Amari · 1998
Earlier work this paper cites.
Efficient backprop
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller · 1998
Earlier work this paper cites.
Adaptive natural gradient learning algorithms for various stochastic models
Hyeyoung Park, Shun-ichi Amari, and Kenji Fukumizu · 2000
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Revisiting natural gradient for deep networks
Original
Razvan Pascanu and Yoshua Bengio · 2014
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy · 2014
Earlier work this paper cites.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015
Earlier work this paper cites.
Riemannian metrics for neural networks I: feedforward networks
Yann Ollivier · 2015
Earlier work this paper cites.
Optimizing neural networks with kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Information geometry and its applications
Shun-ichi Amari · 2016
Earlier work this paper cites.

Similar

Optimal architectures in a solvable model of deep networks
Jonathan Kadmon and Haim Sompolinsky · 2016
Cited alongside, same era.
Exponential expressivity in deep neural networks through transient chaos
Ben Poole, Subhaneil Lahiri, Maithreyi Raghu, Jascha Sohl-Dickstein, and Surya Ganguli · 2016
Cited alongside, same era.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017
Cited alongside, same era.
Empirical analysis of the hessian of over-parametrized neural networks
Original
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou · 2017
Cited alongside, same era.
Deep information propagation
Original
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein · 2017
Cited alongside, same era.
Mean field residual networks: On the edge of chaos
Greg Yang and Samuel Schoenholz · 2017
Cited alongside, same era.
Geometry of neural network loss surfaces via random matrix theory
Jeffrey Pennington and Yasaman Bahri · 2017
Cited alongside, same era.
Nonlinear random matrix theory for deep learning
Jeffrey Pennington and Pratik Worah · 2017
Cited alongside, same era.
The spectrum of the fisher information matrix of a single-hidden-layer neural network
Jeffrey Pennington and Pratik Worah · 2018
Cited alongside, same era.
Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington · 2018
Cited alongside, same era.
The emergence of spectral universality in deep networks
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli · 2018
Cited alongside, same era.

Then

Sensitivity and generalization in neural networks: an empirical study
Original
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein · 2018
Later among the works it cites.
Fast generalization error bound of deep learning from a kernel perspective
Taiji Suzuki · 2018
Later among the works it cites.
Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians
Vardan Papyan · 2019
Closest in time.
An investigation into neural net optimization via hessian eigenvalue density
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao · 2019
Closest in time.
Fisher-rao metric, geometry, and complexity of neural networks
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes · 2019
Closest in time.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation
Original
Greg Yang · 2019
Closest in time.
Wide neural networks of any depth evolve as linear models under gradient descent
Original
Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Closest in time.
On exact computation with an infinitely wide neural net
Original
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang · 2019
Closest in time.
Lightlike neuromanifolds, occam’s razor and deep learning
Original
Ke Sun and Frank Nielsen · 2019
Closest in time.
Fisher information and natural gradient learning of random deep networks
Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi · 2019
Closest in time.
A mean field theory of batch normalization
Original
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz · 2019
Closest in time.
Freeze and chaos for dnns: an NTK view of batch normalization, checkerboard and boundary effects
Original
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2019
Closest in time.
A fine-grained spectral perspective on neural networks
Original
Greg Yang and Hadi Salman · 2019
Closest in time.
The large learning rate phase of deep learning: the catapult mechanism
Original
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…