Fetching the paper…

Reading the bibliography…

2020

Multivariate Gaussian Variational Inference by Natural Gradient Descent

Barfoot, Timothy D.

Original
PDF
Post on X

Understand

This short note reviews so-called Natural Gradient Descent (NGD) for multivariate Gaussians.

The Fisher Information Matrix (FIM) is derived for several different parameterizations of Gaussians.
Careful attention is paid to the symmetric nature of the covariance matrix when calculating derivatives.
We show that there are some advantages to choosing a parameterization comprising the mean and inverse covariance matrix and provide a simple NGD update that accounts for the symmetric (and sparse) nature of the inverse covariance matrix.

Multivariate Gaussian Variational Inference by Natural Gradient Descent · Around

Built on

Barfoot, T. D., Forbes, J. R., and Yoon, D. J., “Exactly Sparse Gaussian Variational Inference with Application to Derivative-Free Batch Nonlinear State Estimation,” 2020, conditionally accepted to the International Journal of Robotics Research (IJRR)
Original
1911
Earlier work this paper cites.
Fisher, R. A., “On the mathematical foundations of theoretical statistics,” Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 222(594-604):309–368, 1922
1922
Earlier work this paper cites.
Kullback, S. and Leibler, R. A., “On information and sufficiency,” The annals of mathematical statistics , 22(1):79–86, 1951
1951
Earlier work this paper cites.

Similar

Magnus, J. R. and Neudecker, H., “The elimination matrix: some lemmas and applications,” SIAM Journal on Algebraic Discrete Methods , 1(4):422–449, 1980
1980
Cited alongside, same era.
Amari, S.-I., “Natural gradient works efficiently in learning,” Neural computation , 10(2):251–276, 1998
1998
Cited alongside, same era.

Then

Opper, M. and Archambeau, C., “The variational Gaussian approximation revisited,” Neural computation , 21(3):786–792, 2009
Original
2009
Later among the works it cites.
Magnus, J. R. and Neudecker, H., Matrix differential calculus with applications in statistics and econometrics , John Wiley & Sons, 2019
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…