An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models

Understand

Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset.

• Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold.
• When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch.
• In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization.