Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimization

Understand

We introduce a hybrid stochastic estimator to design stochastic gradient algorithms for solving stochastic optimization problems.

• Such a hybrid estimator is a convex combination of two existing biased and unbiased estimators and leads to some useful property on its variance.
• We limit our consideration to a hybrid SARAH-SGD for nonconvex expectation problems.
• However, our idea can be extended to handle a broader class of estimators in both convex and nonconvex settings.