Classification vs regression in overparameterized regimes: Does the loss function matter?

Understand

We compare classification and regression tasks in an overparameterized linear model with Gaussian features.

• On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers.
• On the other hand, we show that there exist regimes where these interpolating solutions generalize well when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e.
• they approach the null risk.