Fetching the paper…

Reading the bibliography…

2015

Speed learning on the fly

Massé, Pierre-Yves, Ollivier, Yann

Original
PDF
Post on X

Understand

The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications.

The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad.
Here we propose to adapt the step size by performing a gradient descent on the step size itself, viewing the whole performance of the learning trajectory as a function of step size.
Importantly, this adaptation can be computed online at little cost, without having to iterate backward passes over the full data.

Speed learning on the fly · Around

Built on

A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
Stochastic Approximation Methods for Constrained and Unconstrained Systems
Harold J. Kushner and Dean S. Clark · 1978
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-ichi Amari · 1998
Earlier work this paper cites.
Large-scale machine learning with stochastic gradient descent
Léon Bottou · 2010
Earlier work this paper cites.

Similar

Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Cited alongside, same era.
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Rie Johnson and Tong Zhang · 2013
Cited alongside, same era.
Minimizing finite sums with the stochastic average gradient
Mark Schmidt, Nicolas Le Roux, and Francis Bach · 2013
Cited alongside, same era.

Then

No More Pesky Learning Rates
Original
Tom Schaul, Sixin Zhang, and Yann LeCun · 2013
Later among the works it cites.
SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien · 2014
Later among the works it cites.
Gradient-based hypermarameter optimization through reversible learning
Douglas Maclaurin, David Duvenaud, and Ryan Adams · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…