Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning

Understand

Pretrained language models have achieved state-of-the-art performance when adapted to a downstream NLP task.

• However, theoretical analysis of these models is scarce and challenging since the pretraining and downstream tasks can be very different.
• We propose an analysis framework that links the pretraining and downstream tasks with an underlying latent variable generative model of text -- the downstream classifier must recover a function of the posterior distribution over the latent variables.
• We analyze head tuning (learning a classifier on top of the frozen pretrained model) and prompt tuning in this setting.