Divergence Results and Convergence of a Variance Reduced Version of ADAM

Understand

Stochastic optimization algorithms using exponential moving averages of the past gradients, such as ADAM, RMSProp and AdaGrad, have been having great successes in many applications, especially in training deep neural networks.

• ADAM in particular stands out as efficient and robust.
• Despite of its outstanding performance, ADAM has been proved to be divergent for some specific problems.
• We revisit the divergent question and provide divergent examples under stronger conditions such as in expectation or high probability.