On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions

Understand

The Adaptive Momentum Estimation (Adam) algorithm is highly effective in training various deep learning tasks.

• Despite this, there's limited theoretical understanding for Adam, especially when focusing on its vanilla form in non-convex smooth scenarios with potential unbounded gradients and affine variance noise.
• In this paper, we study vanilla Adam under these challenging conditions.
• We introduce a comprehensive noise model which governs affine variance noise, bounded noise and sub-Gaussian noise.