Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

Understand

Parameter-Efficient Transfer Learning (PETL) aims at efficiently adapting large models pre-trained on massive data to downstream tasks with limited task-specific data.

• In view of the practicality of PETL, previous works focus on tuning a small set of parameters for each downstream task in an end-to-end manner while rarely considering the task distribution shift issue between the pre-training task and the downstream task.
• This paper proposes a novel two-stage paradigm, where the pre-trained model is first aligned to the target distribution.
• Then the task-relevant information is leveraged for effective adaptation.