Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference

Understand

Fine-tuning has been proven to be a simple and effective technique to transfer the learned knowledge of Pre-trained Language Models (PLMs) to downstream tasks.

• However, vanilla fine-tuning easily overfits the target data and degrades the generalization ability.
• Most existing studies attribute it to catastrophic forgetting, and they retain the pre-trained knowledge indiscriminately without identifying what knowledge is transferable.
• Motivated by this, we frame fine-tuning into a causal graph and discover that the crux of catastrophic forgetting lies in the missing causal effects from the pretrained data.