Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond

Understand

Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically.

• Instead, the loss oscillates as gradient descent converges to its ''Edge of Stability'' (EoS).
• Here, we find a quantity that does decrease monotonically throughout GD training: the sharpness attained by the gradient flow solution (GFS)-the solution that would be obtained if, from now until convergence, we train with an infinitesimal step size.
• Theoretically, we analyze scalar neural networks with the squared loss, perhaps the simplest setting where the EoS phenomena still occur.