CLIP model is an Efficient Continual Learner

Understand

The continual learning setting aims to learn new tasks over time without forgetting the previous ones.

• The literature reports several significant efforts to tackle this problem with limited or no access to previous task data.
• Among such efforts, typical solutions offer sophisticated techniques involving memory replay, knowledge distillation, model regularization, and dynamic network expansion.
• The resulting methods have a retraining cost at each learning task, dedicated memory requirements, and setting-specific design choices.