Understand
We provide the first experimental results on non-synthetic datasets for the quasi-diagonal Riemannian gradient descents for neural networks introduced in [Ollivier, 2015].
- These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset.
- The quasi-diagonal Riemannian algorithms consistently beat simple stochastic gradient gradient descents by a varying margin.
- The computational overhead with respect to simple backpropagation is around a factor $2$.
Built on
Untersuchungen zu dynamischen neuronalen Netzen
Sepp Hochreiter · 1991
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-Ichi Amari · 1998
Earlier work this paper cites.
Unsupervised joint alignment of complex images
Gary B. Huang, Vidit Jain, and Erik Learned-Miller · 2007
Earlier work this paper cites.
Topmoumoute online natural gradient algorithm
Nicolas Le Roux, Pierre-Antoine Manzagol, and Yoshua Bengio · 2007
Earlier work this paper cites.
Similar
Adaptive subgradient methods for online learning and stochastic optimization
John C. Duchi, Elad Hazan, and Yoram Singer · 2011
Cited alongside, same era.
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng · 2011
Cited alongside, same era.
Razvan Pascanu and Yoshua Bengio · 2013
Cited alongside, same era.
The MNIST database of handwritten digits
Yann Lecun and Corinna Cortes
Cited in the paper.
Then
New perspectives on the natural gradient method
James Martens · 2014
Later among the works it cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Later among the works it cites.
Riemannian metrics for neural networks I: feedforward networks
Yann Ollivier · 2015
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…