EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

Understand

Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters.

• However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning.
• To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components.
• By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity.