Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

Understand

Pretrained vision-language models (VLMs) like CLIP exhibit exceptional generalization across diverse downstream tasks.

• While recent studies reveal their vulnerability to adversarial attacks, research to date has primarily focused on enhancing the robustness of image encoders against image-based attacks, with defenses against text-based and multimodal attacks remaining largely unexplored.
• To this end, this work presents the first comprehensive study on improving the adversarial robustness of VLMs against attacks targeting image, text, and multimodal inputs.
• This is achieved by proposing multimodal contrastive adversarial training (MMCoA).