The Geometry of Over-parameterized Regression and Adversarial Perturbations

Understand

Classical regression has a simple geometric description in terms of a projection of the training labels onto the column space of the design matrix.

• However, for over-parameterized models -- where the number of fit parameters is large enough to perfectly fit the training data -- this picture becomes uninformative.
• Here, we present an alternative geometric interpretation of regression that applies to both under- and over-parameterized models.
• Unlike the classical picture which takes place in the space of training labels, our new picture resides in the space of input features.