Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization

Understand

A common explanation for the failure of deep networks to generalize out-of-distribution is that they fail to recover the "correct" features.

• We challenge this notion with a simple experiment which suggests that ERM already learns sufficient features and that the current bottleneck is not feature learning, but robust regression.
• Our findings also imply that given a small amount of data from the target distribution, retraining only the last linear layer will give excellent performance.
• We therefore argue that devising simpler methods for learning predictors on existing features is a promising direction for future research.