Optimal Ridge Regularization for Out-of-Distribution Prediction

Understand

We study the behavior of optimal ridge regularization and optimal ridge risk for out-of-distribution prediction, where the test distribution deviates arbitrarily from the train distribution.

• We establish general conditions that determine the sign of the optimal regularization level under covariate and regression shifts.
• These conditions capture the alignment between the covariance and signal structures in the train and test data and reveal stark differences compared to the in-distribution setting.
• For example, a negative regularization level can be optimal under covariate shift or regression shift, even when the training features are isotropic or the design is underparameterized.