Understanding Short-Horizon Bias in Stochastic Meta-Optimization

Understand

Careful tuning of the learning rate, or even schedules thereof, can be crucial to effective neural net training.

• There has been much recent interest in gradient-based meta-optimization, where one tunes hyperparameters, or even learns an optimizer, in order to minimize the expected loss when the training procedure is unrolled.
• But because the training procedure must be unrolled thousands of times, the meta-objective must be defined with an orders-of-magnitude shorter time horizon than is typical for neural net training.
• We show that such short-horizon meta-objectives cause a serious bias towards small step sizes, an effect we term short-horizon bias.