FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

Understand

Large language models (LLMs) have emerged as important components across various fields, yet their training requires substantial computation resources and abundant labeled data.

• It poses a challenge to robustly training LLMs for individual users (clients).
• To tackle this challenge, the intuitive idea is to introduce federated learning (FL), which can collaboratively train models on distributed private data.
• However, existing methods suffer from the challenges of data heterogeneity, system heterogeneity, and model size, resulting in suboptimal performance and high costs.