Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD

Understand

This work considers the problem of finding a first-order stationary point of a non-convex function with potentially unbounded smoothness constant using a stochastic gradient oracle.

• We focus on the class of $(L_0,L_1)$-smooth functions proposed by Zhang et al.
• (ICLR'20).
• Empirical evidence suggests that these functions more closely captures practical machine learning problems as compared to the pervasive $L_0$-smoothness.