Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?

Understand

Methods for learning from data depend on various types of tuning parameters, such as penalization strength or step size.

• Since performance can depend strongly on these parameters, it is important to compare classes of estimators-by considering prescribed finite sets of tuning parameters-not just particularly tuned methods.
• In this work, we investigate classes of methods via the relative performance of the best method in the class.
• We consider the central problem of linear regression-with a random isotropic ground truth-and investigate the estimation performance of two fundamental methods, gradient descent and ridge regression.