Large Continual Instruction Assistant

Understand

Continual Instruction Tuning (CIT) is adopted to continually instruct Large Models to follow human intent data by data.

• It is observed that existing gradient update would heavily destroy the performance on previous datasets during CIT process.
• Instead, Exponential Moving Average (EMA), owns the ability to trace previous parameters, which can aid in decreasing forgetting.
• Nonetheless, its stable balance weight fails to deal with the ever-changing datasets, leading to the out-of-balance between plasticity and stability.