Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator

Understand

Although adaptive optimization algorithms have been successful in many applications, there are still some mysteries in terms of convergence analysis that have not been unraveled.

• This paper provides a novel non-convex analysis of adaptive optimization to uncover some of these mysteries.
• Our contributions are three-fold.
• First, we show that an increasing or large enough momentum parameter for the first-order moment used in practice is sufficient to ensure the convergence of adaptive algorithms whose adaptive scaling factors of the step size are bounded.