Stable Nonconvex-Nonconcave Training via Linear Interpolation

Understand

This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training.

• We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear interpolation can help by leveraging the theory of nonexpansive operators.
• We construct a new optimization scheme called relaxed approximate proximal point (RAPP), which is the first explicit method without anchoring to achieve last iterate convergence rates for $\rho$-comonotone problems while only requiring $\rho > -\tfrac{1}{2L}$.
• The construction extends to constrained and regularized settings.