DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution

Understand

Fine-tuning large-scale pre-trained models is inherently a resource-intensive task.

• While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks.
• Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes.
• To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method.