Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

Understand

Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training.

• Nevertheless, during the continual training of the Contrastive Language-Image Pre-training (CLIP) model, we observe that the model's zero-shot transfer ability significantly degrades due to catastrophic forgetting.
• Existing CL methods can mitigate forgetting by replaying previous data.
• However, since the CLIP dataset is private, replay methods cannot access the pre-training dataset.