Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimization

J. Drenth, “Principles of protein x-ray crystallography,” Springer Science & Business Media

1994

Earlier work this paper cites.

J. Miao, P. Charalambous, J. Kirz, and D. Sayre, “Extending the methodology of x-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens,” Nature

Original

1999

Earlier work this paper cites.

Y. Nesterov and B. Polyak, “Cubic regularization of newton’s method and its global performance,” Mathematical Programming

2006

Earlier work this paper cites.

S. Ghadimi and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic programming,” SIAM Journal on Optimization

2013

Earlier work this paper cites.

R. Johnson and T. Zhang, “Accelerating stochastic gradient descent using predictive variance reduction,” Advances in neural information processing systems

2013

Earlier work this paper cites.

S. J. Reddi, A. Hefny, S. Sra, B. Póczos, and A. Smola, “Stochastic variance reduction for nonconvex optimization,” in International conference on machine learning

2016

Earlier work this paper cites.

L. M. Nguyen, J. Liu, K. Scheinberg, and M. Takáč, “Sarah: A novel method for machine learning problems using stochastic recursive gradient,” in International Conference on Machine Learning

2017

Earlier work this paper cites.

C. Fang, C. J. Li, Z. Lin, and T. Zhang, “Spider: Near-optimal non-convex optimization via stochastic path-integrated differential estimator,” Advances in Neural Information Processing Systems

2018

Earlier work this paper cites.

Z. Wang, K. Ji, Y. Zhou, Y. Liang, and V. Tarokh, “Spiderboost and momentum: Faster variance reduction algorithms,” Advances in Neural Information Processing Systems

2019

Cited alongside, same era.

A. Cutkosky and F. Orabona, “Momentum-based variance reduction in non-convex sgd,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems

2019

Cited alongside, same era.

J. Zhang, T. He, S. Sra, and A. Jadbabaie, “Why gradient clipping accelerates training: A theoretical justification for adaptivity,” in International Conference on Learning Representations

2019

Cited alongside, same era.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford, “Lower bounds for finding stationary points i,” Mathematical Programming

2020

Cited alongside, same era.

D. Levy, Y. Carmon, J. C. Duchi, and A. Sidford, “Large-scale methods for distributionally robust optimization,” Advances in Neural Information Processing Systems

J. Jin, B. Zhang, H. Wang, and L. Wang, “Non-convex distributionally robust optimization: Non-asymptotic analysis,” Advances in Neural Information Processing Systems

2021

Later among the works it cites.

S.-Y. Zhao, Y.-P. Xie, and W.-J. Li, “On the convergence and improvement of stochastic normalized gradient descent,” Science China Information Sciences

2021

Later among the works it cites.

Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford, “Lower bounds for finding stationary points ii: first-order methods,” Mathematical Programming

2021

Later among the works it cites.

Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. Woodworth, “Lower bounds for non-convex stochastic optimization,” Mathematical Programming

2022

Later among the works it cites.

X. Yang, H. Zhang, W. Chen, and T.-Y. Liu, “Normalized/clipped sgd with perturbation for differentially private non-convex optimization,” ArXiv:2206.13033