On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Understand

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions.

• Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes.
• This has motivated a large body of research on adaptive stepsizes.
• However, there is currently a gap in our theoretical understanding of these methods, especially in the non-convex setting.