AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

Understand

Large pretrained language models are widely used in downstream NLP tasks via task-specific fine-tuning, but such procedures can be costly.

• Recently, Parameter-Efficient Fine-Tuning (PEFT) methods have achieved strong task performance while updating much fewer parameters than full model fine-tuning (FFT).
• However, it is non-trivial to make informed design choices on the PEFT configurations, such as their architecture, the number of tunable parameters, and even the layers in which the PEFT modules are inserted.
• Consequently, it is highly likely that the current, manually designed configurations are suboptimal in terms of their performance-efficiency trade-off.