Fetching the paper…

Reading the bibliography…

2022

Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates

Puiu, Constantin Octavian

Original
PDF
Post on X

Understand

K-FAC (arXiv:1503.05671, arXiv:1602.01407) is a tractable implementation of Natural Gradient (NG) for Deep Learning (DL), whose bottleneck is computing the inverses of the so-called ``Kronecker-Factors'' (K-factors).

RS-KFAC (arXiv:2206.15397) is a K-FAC improvement which provides a cheap way of estimating the K-factors inverses.
In this paper, we exploit the exponential-average construction paradigm of the K-factors, and use online numerical linear algebra techniques to propose an even cheaper (but less accurate) way of estimating the K-factors inverses.
In particular, we propose a K-factor inverse update which scales linearly in layer size.

Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates · Around

Built on

Amari, S. I. Natural gradient works efficiently in learning, Neural Computation, 10(20), pp. 251-276 (1998)
1998
Earlier work this paper cites.
Brand, M. Fast Low-Rank Modifications of the Thin Singular Value Decomposition, Linear Algebra and its Applications Volume 415, Issue 1, pp. 20-30, 2006
2006
Earlier work this paper cites.
Halko N.; Martinsson P.G.; Tropp J. A. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, SIAM Review
2011
Earlier work this paper cites.
Martens, J.; Grosse, R. Optimizing neural networks with Kronecker-factored approximate curvature, In: arXiv:1503.05671
Original
2015
Earlier work this paper cites.

Similar

Grosse, R.; Martens J. A Kronecker-factored approximate Fisher matrix for convolution layers, arXiv:1602.01407 (2016)
Original
2016
Cited alongside, same era.
Saibaba, A. K. Randomized subspace iteration: Analysis of canonical angles and unitarily invariant norms, arXiv:1804.02614 (2018)
Original
2018
Cited alongside, same era.
Martens, J. New insights and perspectives on the natural gradient method, arXiv:1412.1193 (2020)
Original
2020
Cited alongside, same era.
Osawa, K.; Yuichiro Ueno, T.; Naruse, A.; Foo, C.-S.; Yokota, R. Scalable and practical natural gradient for large-scale deep learning, arXiv:2002.06015 (2020)
Original
2020
Cited alongside, same era.
Mazeika M. The Singular Value Decomposition and Low Rank Approximation
Original
Cited in the paper.

Then

Yang, M.; Xu, D; Wen, Z.; Chen, M.; Xu, P. Sketchy empirical natural gradient methods for deep learning, arXiv:2006.05924 (2021)
Original
2021
Later among the works it cites.
Tang, Z.; Jiang, F.; Gong, M.; Li, H.; Wu, Y.; Yu, F.; Wang, Z.; Wang, M. SKFAC: Training Neural Networks with Faster Kronecker-Factored Approximate Curvature, IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021)
2021
Later among the works it cites.
Puiu, C. O. Randomized KFACs: Speeding up K-FAC with Randomized Numerical Linear Algebra, arXiv:2206.15397 (2022)
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…