Special Properties of Gradient Descent with Large Learning Rates

Understand

When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models.

• However, the effect of large step sizes on the success of SGD is not well understood theoretically.
• Several previous works have attributed this success to the stochastic noise present in SGD.
• However, we show through a novel set of experiments that the stochastic noise is not sufficient to explain good non-convex training, and that instead the effect of a large learning rate itself is essential for obtaining best performance.We demonstrate the same effects also in the noise-less case, i.e.