New Convergence Aspects of Stochastic Gradient Algorithms

Understand

The classical convergence analysis of SGD is carried out under the assumption that the norm of the stochastic gradient is uniformly bounded.

• While this might hold for some loss functions, it is violated for cases where the objective function is strongly convex.
• In Bottou et al.
• (2018), a new analysis of convergence of SGD is performed under the assumption that stochastic gradients are bounded with respect to the true gradient norm.