Krylov Subspace Descent for Deep Learning

Fast exact multiplication by the Hessian

Barak A. Pearlmutter · 1994

Earlier work this paper cites.

Natural gradient works efficiently in learning

Shun-Ichi Amari · 1998

Earlier work this paper cites.

Enriched Methods for Large-Scale Unconstrained Optimization

José Luis Morales and Jorge Nocedal · 2000

Earlier work this paper cites.

Fast curvature matrix-vector products for second-order gradient descent

Nicol N. Schraudolph · 2002

Earlier work this paper cites.

Reducing the Dimensionality of Data with Neural Networks

Geoffrey Hinton and Ruslan Salakhutdinov · 2006

Earlier work this paper cites.

Numerical Optimization

Jorge Nocedal and Stephen J. Wright · 2006

Cited alongside, same era.

Topmoumoute online natural gradient algorithm

Nicolas Le Roux, Yoshua Bengio, and Pierre antoine Manzagol · 2007

Cited alongside, same era.

The difficulty of training deep architectures and the effect of unsupervised pre-training

Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent · 2009

Cited alongside, same era.

Understanding the difficulty of training deep feedforward neural networks

Yoshua Bengio and Xavier Glorot · 2010

Cited alongside, same era.

On the use of stochastic hessian information in optimization methods for machine learning

Richard H. Byrd, Gillian M. Chiny, Will Neveitt, and Jorge Nocedal · 2010

Cited alongside, same era.

Deep learning via Hessian-free optimization

James Martens · 2010

Later among the works it cites.

On optimization methods for deep learning

Quoc V. Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y. Ng · 2011

Closest in time.

Learning Recurrent Neural Networks with Hessian-Free Optimization

James Martens and Ilya Sutskever · 2011

Closest in time.

Comparing Multilayer Perceptron to Deep Belief Network Tandem Features for Robust ASR

Oriol Vinyals and Suman Ravuri · 2011

Closest in time.