Catapult Dynamics and Phase Transitions in Quadratic Nets

Understand

Neural networks trained with gradient descent can undergo non-trivial phase transitions as a function of the learning rate.

• In \cite{lewkowycz2020large} it was discovered that wide neural nets can exhibit a catapult phase for super-critical learning rates, where the training loss grows exponentially quickly at early times before rapidly decreasing to a small value.
• During this phase the top eigenvalue of the neural tangent kernel (NTK) also undergoes significant evolution.
• In this work, we will prove that the catapult phase exists in a large class of models, including quadratic models and two-layer, homogenous neural nets.