Convergence Analysis of Homotopy-SGD for non-convex optimization

Understand

First-order stochastic methods for solving large-scale non-convex optimization problems are widely used in many big-data applications, e.g.

• training deep neural networks as well as other complex and potentially non-convex machine learning models.
• Their inexpensive iterations generally come together with slow global convergence rate (mostly sublinear), leading to the necessity of carrying out a very high number of iterations before the iterates reach a neighborhood of a minimizer.
• In this work, we present a first-order stochastic algorithm based on a combination of homotopy methods and SGD, called Homotopy-Stochastic Gradient Descent (H-SGD), which finds interesting connections with some proposed heuristics in the literature, e.g.