MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

Understand

LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks.

• Since ChatGPT demonstrated superior performance on various tasks, there has been a growing desire to adapt one model for all tasks.
• However, the explicit low-rank of LoRA limits the adaptation performance in complex multi-task scenarios.
• LoRA is dominated by a small number of top singular vectors while fine-tuning decomposes into a set of less important unitary transforms.