Tighter Analysis of Alternating Stochastic Gradient Method for Stochastic Nested Problems

Understand

Stochastic nested optimization, including stochastic compositional, min-max and bilevel optimization, is gaining popularity in many machine learning applications.

• While the three problems share the nested structure, existing works often treat them separately, and thus develop problem-specific algorithms and their analyses.
• Among various exciting developments, simple SGD-type updates (potentially on multiple variables) are still prevalent in solving this class of nested problems, but they are believed to have slower convergence rate compared to that of the non-nested problems.
• This paper unifies several SGD-type updates for stochastic nested problems into a single SGD approach that we term ALternating Stochastic gradient dEscenT (ALSET) method.