CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

Understand

Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks.

• This poses significant risks to downstream model training.
• In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data.
• In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance.