Fisher Information and Natural Gradient Learning of Random Deep Networks

S. Amari, Theory of Adaptive Pattern Classifiers, IEEE Trans., EC-16, No. 3, pp.299–307, 1967

1967

Earlier work this paper cites.

S. Amari, Information Theory II, Geometrical Theory of Information (in Japanese), Kyoritsu Shuppan, 1968

1968

Earlier work this paper cites.

B.T. Polyak and A.B. Juditsky, Acceleration of stochastic approximation by averaging. SIAM J. on Control and Optimization, 30, 838-855, 1992

1992

Earlier work this paper cites.

T. Kurita, Iterative weighted least squares algorithms for neural networks classifiers. New Generation Computing, 12, 3750394, 1994

1994

Earlier work this paper cites.

S. Amari, Natural Gradient Works Efficiently in Learning, Neural Computation, Vol. 10, No. 2, pp. 251–276, 1998

1998

Earlier work this paper cites.

R. Pascanu and Y. Bengio, Revisiting natural gradient for deep networks. arXiv: 1301.3584v7, 2013

2013

Earlier work this paper cites.

Y. Ollivier, Riemannian metrics for neural networks, I: Feedforward networks. Information and Inference, 4, 108–153, 2015

2015

Cited alongside, same era.

R. Grosse and J. Martens, A Kronecker-factored approximate Fisher matrix for convolution layers, ICML, 2016

2016

Cited alongside, same era.

G. Marceau-Caron and Y. Ollivier, Practical Riemannian neural networks, arXiv:1602.08007, 2016

Original

2016

Cited alongside, same era.

B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein and S. Ganguli, Exponential expressivity in deep neural networks. In Proc. NIPS, 3360–3368, 2016

2016

Cited alongside, same era.

S. S. Schoenholz, J. Gilmer, S. Ganguli and J Sohl-Dickstein, Deep information propagation. ICLR’2017, arXiv: 1611.01232, 2016

2016

Cited alongside, same era.

J. Martens, New insight and perspective on the natural gradient method, arXiv:1412.1193v9, 2017

Original

2017

Later among the works it cites.

Y. Ollivier, True asymptotic natural gradient optimization. arXiv: 1712.08449v1, 2017

2017

Later among the works it cites.

G. Yang, S. Schoenholz, Mean field residual networks: On the edge of chaos. Proc. NIPS, 2865–2873, 2017

2017

Later among the works it cites.

S. Amari, R. Karakida and M. Oizumi, Statistical neurodynamics of deep networks I, Geometry of signal spaces. arXiv, 2018

2018

Closest in time.

R. Karakida, S. Akaho and S. Amari, Universal statistics of Fisher information in deep neural networks: Mean field approach, arXiv.org/abs/1806.01316, 2018

Original

2018

Closest in time.