A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

Understand

As large language models (LLMs) have become increasingly compute and memory intensive, parameter-efficient fine-tuning (PEFT) methods are now a common strategy to fine-tune LLMs.

• A popular PEFT method is Low-Rank Adapters (LoRA), which adds trainable low-rank "adapters" to selected layers.
• Each adapter consists of a low-rank matrix product, multiplicatively scaled by a rank-dependent factor.
• This scaling factor, which divides adapters by a factor of the rank, results in slowed learning and stunted performance for LoRA with higher-rank adapters.