Small steps and giant leaps: Minimal Newton solvers for Deep Learning

Understand

We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers.

• Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement.
• Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, a procedure that is both costly and sensitive to noise.
• Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration.