A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization

Understand

Machine learning practitioners invest significant manual and computational resources in finding suitable learning rates for optimization algorithms.

• We provide a probabilistic motivation, in terms of Gaussian inference, for popular stochastic first-order methods.
• As an important special case, it recovers the Polyak step with a general metric.
• The inference allows us to relate the learning rate to a dimensionless quantity that can be automatically adapted during training by a control algorithm.