Parameter-Efficient Fine-Tuning Design Spaces

Understand

Parameter-efficient fine-tuning aims to achieve performance comparable to fine-tuning, using fewer trainable parameters.

• Several strategies (e.g., Adapters, prefix tuning, BitFit, and LoRA) have been proposed.
• However, their designs are hand-crafted separately, and it remains unclear whether certain design patterns exist for parameter-efficient fine-tuning.
• Thus, we present a parameter-efficient fine-tuning design paradigm and discover design patterns that are applicable to different experimental settings.