Rethinking Why Intermediate-Task Fine-Tuning Works

Understand

Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest.

• While STILTs is able to further improve the performance of pretrained language models, it is still unclear why and when it works.
• Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa.
• In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills -- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks.