The Impact of Local Geometry and Batch Size on Stochastic Gradient Descent for Nonconvex Problems

M. Styblinski and T.S. Tang, Experiments in nonconvex optimization: stochastic approximation with function smoothing and simulated annealing , Neural Networks 3 (1990), pp. 467–483

1990

Earlier work this paper cites.

S. Hochreiter and J. Schmidhuber, Simplifying neural nets by discovering flat minima , in Advances in neural information processing systems . 1995, pp. 529–536

1995

Earlier work this paper cites.

D.P. Bertsekas, Nonlinear programming , Athena scientific Belmont, 1999

1999

Earlier work this paper cites.

R. Ge, F. Huang, C. Jin, and Y. Yuan, Escaping from saddle points—online stochastic gradient for tensor decomposition , in Conference on Learning Theory . 2015, pp. 797–842

2015

Earlier work this paper cites.

M. Hardt, B. Recht, and Y. Singer, Train faster, generalize better: Stability of stochastic gradient descent , arXiv preprint arXiv:1509.01240 (2015)

Original

2015

Earlier work this paper cites.

L. Bottou, F.E. Curtis, and J. Nocedal, Optimization methods for large-scale machine learning , arXiv preprint arXiv:1606.04838 (2016)

Original

2016

Cited alongside, same era.

P. Chaudhari, A. Choromanska, S. Soatto, and Y. LeCun, Entropy-sgd: Biasing gradient descent into wide valleys , arXiv preprint arXiv:1611.01838 (2016)

Original

2016

Cited alongside, same era.

N.S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P.T.P. Tang, On large-batch training for deep learning: Generalization gap and sharp minima , arXiv preprint arXiv:1609.04836 (2016)

Original

2016

Cited alongside, same era.

V. Patel, Kalman-based stochastic gradient method with stop condition and insensitivity to conditioning , SIAM Journal on Optimization 26 (2016), pp. 2620–2648

2016

Cited alongside, same era.

P. Balaprakash, Private Communication (2017)

L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio, Sharp minima can generalize for deep nets , arXiv preprint arXiv:1703.04933 (2017)

Original

2017

Closest in time.

C. Jin, R. Ge, P. Netrapalli, S.M. Kakade, and M.I. Jordan, How to escape saddle points efficiently , arXiv preprint arXiv:1703.00887 (2017)

Original

2017

Closest in time.

W. Mou, L. Wang, X. Zhai, and K. Zheng, Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints , arXiv preprint arXiv:1707.05947 (2017)

Original

2017

Closest in time.

Y. Zhang, P. Liang, and M. Charikar, A hitting time analysis of stochastic gradient langevin dynamics , arXiv preprint arXiv:1702.05575 (2017)

Original

2017

Closest in time.

R. Kleinberg, Y. Li, and Y. Yuan, An alternative view: When does sgd escape local minima? , arXiv preprint arXiv:1802.06175 (2018)