Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis

Understand

Stochastic gradient descent (SGD) is one of the most popular algorithms in modern machine learning.

• The noise encountered in these applications is different from that in many theoretical analyses of stochastic gradient algorithms.
• In this article, we discuss some of the common properties of energy landscapes and stochastic noise encountered in machine learning problems, and how they affect SGD-based optimization.
• In particular, we show that the learning rate in SGD with machine learning noise can be chosen to be small, but uniformly positive for all times if the energy landscape resembles that of overparametrized deep learning problems.