Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization

Understand

Two new stochastic variance-reduced algorithms named SARAH and SPIDER have been recently proposed, and SPIDER has been shown to achieve a near-optimal gradient oracle complexity for nonconvex optimization.

• However, the theoretical advantage of SPIDER does not lead to substantial improvement of practical performance over SVRG.
• To address this issue, momentum technique can be a good candidate to improve the performance of SPIDER.
• However, existing momentum schemes used in variance-reduced algorithms are designed specifically for convex optimization, and are not applicable to nonconvex scenarios.