Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noise

Understand

The existence of spurious correlations such as image backgrounds in the training environment can make empirical risk minimization (ERM) perform badly in the test environment.

• To address this problem, Kirichenko et al.
• (2022) empirically found that the core features that are related to the outcome can still be learned well even with the presence of spurious correlations.
• This opens a promising strategy to first train a feature learner rather than a classifier, and then perform linear probing (last layer retraining) in the test environment.