On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm

Understand

The pretraining-finetuning paradigm has become the prevailing trend in modern deep learning.

• In this work, we discover an intriguing linear phenomenon in models that are initialized from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL).
• Specifically, we show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linear interpolation of features in two finetuned models at each layer.
• We provide comprehensive empirical evidence supporting that CTL consistently occurs for finetuned models that start from the same pretrained checkpoint.