Adaptive Step Sizes in Variance Reduction via Regularization

Understand

The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size.

• With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex.
• To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex ones via regularization.
• The proposed regularization schemes are \textit{simple} yet effective.