The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use

Built on

Differential Geometry in Statistical Inference
Amari, S.-I. (1987) · 1987
Earlier work this paper cites.
A variational approach to Bayesian logistic regression models and their extensions
Jaakkola, T. and Jordan, M. (1997) · 1997
Earlier work this paper cites.
Mean field approach to learning in Boltzmann Machines
Kappen, H. and Rodriguez, F. (1997) · 1997
Earlier work this paper cites.
Natural Gradient Works Efficiently in Learning
Amari, S.-I. (1998) · 1998
Earlier work this paper cites.
Mean-field theory of Boltzmann machine learning
Tanaka, T. (1998) · 1998
Earlier work this paper cites.

Methods of Information Geometry
Amari, S. and Nagaoka, H. (2000) · 2000
Cited alongside, same era.
Training products of experts by minimizing contrastive divergence
Hinton, G. E. (2002) · 2002
Cited alongside, same era.
A new learning algorithm for mean field Boltzmann machines
Original
Welling, M. and Hinton, G. (2002) · 2002
Cited alongside, same era.
Convex optimization
Boyd, S. P. and Vandenberghe, L. (2004) · 2004
Cited alongside, same era.
New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow
Sohl-Dickstein, J., Battaglino, P., and DeWeese, M. (2011a)
Cited in the paper.
Minimum Probability Flow Learning
Sohl-Dickstein, J., Battaglino, P. B., and DeWeese, M. R. (2011b)
Cited in the paper.

Estimation of non-normalized statistical models using score matching
Hyvärinen, A. (2005) · 2005
Later among the works it cites.
Gradients on matrix manifolds and their chain rule
Theis, F. (2005) · 2005
Later among the works it cites.
Neural networks and learning machines; 3rd edition
Haykin, S. (2008) · 2008
Later among the works it cites.
Information geometry in optimization, machine learning and statistical inference
Amari, S.-i. (2010) · 2010
Later among the works it cites.