Randomized K-FACs: Speeding up K-FAC with Randomized Numerical Linear Algebra

Amari, S. I. Natural gradient works efficiently in learning, Neural Computation, 10(20), pp. 251-276 (1998)

1998

Earlier work this paper cites.

Halko N.; Martinsson P.G.; Tropp J. A. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions (2011)

2011

Earlier work this paper cites.

Martens, J.; Grosse, R. Optimizing neural networks with Kronecker-factored approximate curvature, arXiv:1503.05671 (2015)

Original

2015

Earlier work this paper cites.

Grosse, R.; Martens J. A Kronecker-factored approximate Fisher matrix for convolution layers, arXiv:1602.01407 (2016)

Original

2016

Earlier work this paper cites.

Tropp, J. A.; Yurtsever, A.; Udell, M.; Cevher, V. Practical Sketching Algorithms for Low-Rank Matrix approximation, arXiv:1609.00048 (2017)

Original

2017

Earlier work this paper cites.

Saibaba, A. K. Randomized subspace iteration: Analysis of canonical angles and unitarily invariant norms, arXiv:1804.02614 (2018)

Original

2018

Cited alongside, same era.

Martens, J. New insights and perspectives on the natural gradient method, arXiv:1412.1193 (2020)

Original

2020

Cited alongside, same era.

Osawa, K.; Yuichiro Ueno, T.; Naruse, A.; Foo, C.-S.; Yokota, R. Scalable and practical natural gradient for large-scale deep learning, arXiv:2002.06015 (2020)

Original

2020

Cited alongside, same era.

Gao, K.-X.; Liu X.-L.; Huang Z.-H.; Wang, M.; Want S.; Wang, Z.; Xu, D.; Yu, F. Eigenvalue-corrected NG Based on a New Approximation, arXiv:2011.13609 (2020)

Original

2020

Cited alongside, same era.

Mazeika M. The Singular Value Decomposition and Low Rank Approximation

Yang, M.; Xu, D; Wen, Z.; Chen, M.; Xu, P. Sketchy empirical natural gradient methods for deep learning, arXiv:2006.05924 (2021)

Original

2021

Later among the works it cites.

Tang, Z.; Jiang, F.; Gong, M.; Li, H.; Wu, Y.; Yu, F.; Wang, Z.; Wang, M. SKFAC: Training Neural Networks with Faster Kronecker-Factored Approximate Curvature, IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021)

2021

Later among the works it cites.

Murray, M.; Abrol, V.; Tanner, J. Activation function design for deep networks: linearity and effective initialisation, in arXiv:2105.07741, (2021)

Original

2021

Later among the works it cites.

Puiu, C. O. Rethinking Exponential Averaging of the Fisher, arXiv:2204.04718 (2022)

Original

2022

Closest in time.