On the Effectiveness of Parameter-Efficient Fine-Tuning

Understand

Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks.

• However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task.
• Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks.
• These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts.