Fetching the paper…
Reading the bibliography…
We prove that the norm version of the adaptive stochastic gradient method (AdaGrad-Norm) achieves a linear convergence rate for a subset of either strongly convex functions or non-convex functions that satisfy the Polyak Lojasiewicz (PL) inequality.
On the convergence of adam and beyond
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar · 1904
Earlier work this paper cites.
Une propriété topologique des sous-ensembles analytiques réels
Stanislaw Lojasiewicz · 1963
Earlier work this paper cites.
Gradient methods for minimizing functionals
Boris Teodorovich Polyak · 1963
Earlier work this paper cites.
Une Approche théorique de l’Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole
Léon Bottou · 1991
Earlier work this paper cites.
A numerical study of the limited memory bfgs method and the truncated-newton method for large scale optimization
Stephen G Nash and Jorge Nocedal · 1991
Earlier work this paper cites.
Nonlinear programming
Dimitri P Bertsekas · 1999
Earlier work this paper cites.
Large scale online learning
Léon Bottou and Yann L Cun · 2004
Earlier work this paper cites.
Smooth minimization of non-smooth functions
Yu Nesterov · 2005
Earlier work this paper cites.
Cognitive radio: brain-empowered wireless communications
Simon Haykin et al · 2005
Earlier work this paper cites.
Adaptive bound optimization for online convex optimization
H Brendan McMahan and Matthew Streeter · 2010
Earlier work this paper cites.
Non-asymptotic analysis of stochastic approximation algorithms for machine learning
Eric Moulines and Francis R Bach · 2011
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
A stochastic gradient method with an exponential convergence _rate for finite training sets
Nicolas L. Roux, Mark Schmidt, and Francis R. Bach · 2012
Earlier work this paper cites.
Accelerating stochastic gradient descent using predictive variance reduction
Rie Johnson and Tong Zhang · 2013
Earlier work this paper cites.
Fast convergence of stochastic gradient descent under a strong growth condition
Mark Schmidt and Nicolas Le Roux · 2013
Earlier work this paper cites.
Saga: A fast incremental gradient method with support for non-strongly convex composite objectives
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Convex optimization: Algorithms and complexity
Sébastien Bubeck et al · 2015
Cited alongside, same era.
Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm
Deanna Needell, Nathan Srebro, and Rachel Ward · 2016
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Cited alongside, same era.
Online adaptive methods, universality and acceleration
Yehuda Kfir Levy, Alp Yurtsever, and Volkan Cevher · 2018
Later among the works it cites.
On the convergence of stochastic gradient descent with adaptive stepsizes
Xiaoyu Li and Francesco Orabona · 2018
Later among the works it cites.
Dissecting adam: The sign, magnitude and variance of stochastic gradients
Lukas Balles and Philipp Hennig · 2018
Later among the works it cites.
signSGD: Compressed Optimisation for Non-Convex Problems
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar · 2018
Later among the works it cites.
SADAGRAD: Strongly adaptive stochastic gradient methods
Zaiyi Chen, Yi Xu, Enhong Chen, and Tianbao Yang · 2018
Later among the works it cites.
Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Minimizing finite sums with the stochastic average gradient
Mark Schmidt, Nicolas Le Roux, and Francis Bach · 2017
Cited alongside, same era.
Diagonal rescaling for neural networks
Jean Lafond, Nicolas Vasilache, and Léon Bottou · 2017
Cited alongside, same era.
Online to offline conversions, universality and adaptive minibatch sizes
Kfir Levy · 2017
Cited alongside, same era.
Variants of rmsprop and adagrad with logarithmic regret bounds
Mahesh Chandra Mukkamala and Matthias Hein · 2017
Cited alongside, same era.
Convergence analysis of two-layer neural networks with relu activation
Yuanzhi Li and Yang Yuan · 2017
Cited alongside, same era.
Optimization methods for large-scale machine learning
Léon Bottou, Frank E Curtis, and Jorge Nocedal · 2018
Cited alongside, same era.
A convergence theory for deep learning via over-parameterization
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song · 2018
Cited alongside, same era.
Sharan Vaswani, Francis Bach, and Mark Schmidt · 2018
Later among the works it cites.
On exponential convergence of sgd in non-convex over-parametrized learning
Raef Bassily, Mikhail Belkin, and Siyuan Ma · 2018
Later among the works it cites.
An alternative view: When does sgd escape local minima?
Robert Kleinberg, Yuanzhi Li, and Yang Yuan · 2018
Later among the works it cites.
Escaping saddle points with adaptive gradient methods
Matthew Staib, Sashank J Reddi, Satyen Kale, Sanjiv Kumar, and Suvrit Sra · 2019
Closest in time.
Gradient descent provably optimizes over-parameterized neural networks
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh · 2019
Closest in time.
SGD converges to global minimum in deep learning via star-convex path
Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh · 2019
Closest in time.
On the linear convergence of the stochastic gradient method with constant step-size
Volkan Cevher and Bằng Công Vũ · 2019
Closest in time.
High-dimensional statistics: A non-asymptotic viewpoint , volume 48
Martin J Wainwright · 2019
Closest in time.
Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee · 2019
Closest in time.
Global convergence of adaptive gradient methods for an over-parameterized neural network
Xiaoxia Wu, Simon S Du, and Rachel Ward · 2019
Closest in time.