Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Built on

Some methods of speeding up the convergence of iteration methods
B. T. Polyak · 1964
Earlier work this paper cites.
A method of solving a convex programming problem with convergence rate O(1/sqr(k))
Yurii Nesterov · 1983
Earlier work this paper cites.
Introductory lectures on convex optimization : a basic course
Yurii Nesterov · 2004
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.

Stochastic first- and zeroth-order methods for nonconvex stochastic programming
Saeed Ghadimi and Guanghui Lan · 2013
Cited alongside, same era.
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton · 2013
Cited alongside, same era.
Global convergence of the heavy-ball method for convex optimization
Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson · 2014
Cited alongside, same era.
ipiano: Inertial proximal algorithm for nonconvex optimization
Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock · 2014
Cited alongside, same era.

ipiasco: Inertial proximal algorithm for strongly convex optimization
Peter Ochs, Thomas Brox, and Thomas Pock · 2015
Later among the works it cites.
Accelerated gradient methods for nonconvex nonlinear and stochastic programming
Saeed Ghadimi and Guanghui Lan · 2016
Closest in time.
Stochastic variance reduction for nonconvex optimization
Original
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alexander J. Smola · 2016
Closest in time.
Variance reduction for faster non-convex optimization
Original
Zeyuan Allen Zhu and Elad Hazan · 2016
Closest in time.