Fetching the paper…

Reading the bibliography…

2022

Rethinking Exponential Averaging of the Fisher

Puiu, Constantin Octavian

Original
PDF
Post on X

Understand

In optimization for Machine learning (ML), it is typical that curvature-matrix (CM) estimates rely on an exponential average (EA) of local estimates (giving EA-CM algorithms).

This approach has little principled justification, but is very often used in practice.
In this paper, we draw a connection between EA-CM algorithms and what we call a "Wake of Quadratic regularized models".
The outlined connection allows us to understand what EA-CM algorithms are doing from an optimization perspective.

Rethinking Exponential Averaging of the Fisher · Around

Built on

LeCun, Y.; Bottou, L.; Orr, G.; Muller, K. Efficient backprop. Neural networks: Tricks of the trade, pages 546-546 (1998)
1998
Earlier work this paper cites.
Amari, S. I. Natural gradient works efficiently in learning, Neural Computation, 10(20), pp. 251-276 (1998)
1998
Earlier work this paper cites.
Park, H.; Amari, S.-I.; Fukumizu, K. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13(7):755-764 (2000)
2000
Earlier work this paper cites.
Schaul, T.; Zhang, S.; LeCun, Y. No more pesky learning rates. In ICML (2013)
2013
Earlier work this paper cites.

Similar

Martens, J.; Grosse, R. Optimizing neural networks with Kronecker-factored approximate curvature, In: arXiv:1503.05671 (2015)
Original
2015
Cited alongside, same era.
Ba, J.; Kingma, D. Adam: A method for stochastic optimization, ICLR (2015)
2015
Cited alongside, same era.
Wu, Y.; Mansimov, E.; Grosse, R. B.; Liao, S.; Ba, J. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in neural information processing systems, pages 5285-5294 (2017)
2017
Cited alongside, same era.
Bottou, L.; Curtis, F. E.; Nocedal, J. Optimization methods for large-scale machine learning (2018)
2018
Cited alongside, same era.

Then

Martens, J. New insights and perspectives on the natural gradient method, arXiv:1412.1193 (2020)
Original
2020
Later among the works it cites.
Osawa, K.; Yuichiro Ueno, T.; Naruse, A.; Foo, C.-S.; Yokota, R. Scalable and practical natural gradient for large-scale deep learning, arXiv:2002.06015 (2020)
Original
2020
Later among the works it cites.
Yang, M.; Xu, D; Wen, Z.; Chen, M.; Xu, P. Sketchy empirical natural gradient methods for deep learning, In: arXiv:2006.05924 (2021)
Original
2021
Later among the works it cites.
Goldfarb, D.; Ren, Y.; Bahamou, A. Practical Quasi-Newton methods for training deep neural networks, arXiv:2006.08877, (2021)
Original
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…