A Continuous-Time View of Early Stopping for Least Squares

Understand

We study the statistical properties of the iterates generated by gradient descent, applied to the fundamental problem of least squares regression.

• We take a continuous-time view, i.e., consider infinitesimal step sizes in gradient descent, in which case the iterates form a trajectory called gradient flow.
• Our primary focus is to compare the risk of gradient flow to that of ridge regression.
• Under the calibration $t=1/\lambda$---where $t$ is the time parameter in gradient flow, and $\lambda$ the tuning parameter in ridge regression---we prove that the risk of gradient flow is no less than 1.69 times that of ridge, along the entire path (for all $t \geq 0$).