FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learning

Understand

How to adapt a pre-trained model continuously for sequential tasks with different prediction class labels and domains and finally learn a generalizable model across diverse tasks is a long-lasting challenge.

• Continual learning (CL) has emerged as a promising approach to leverage pre-trained models (e.g., Transformers) for sequential tasks.
• While many existing CL methods incrementally store additional learned structures, such as Low-Rank Adaptation (LoRA) adapters or prompts and sometimes even preserve features from previous samples to maintain performance.
• This leads to unsustainable parameter growth and escalating storage costs as the number of tasks increases.