A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models

Understand

With Vision-Language Pre-training (VLP) models demonstrating powerful multimodal interaction capabilities, the application scenarios of neural networks are no longer confined to unimodal domains but have expanded to more complex multimodal V+L downstream tasks.

• The security vulnerabilities of unimodal models have been extensively examined, whereas those of VLP models remain challenging.
• We note that in CV models, the understanding of images comes from annotated information, while VLP models are designed to learn image representations directly from raw text.
• Motivated by this discrepancy, we developed the Feature Guidance Attack (FGA), a novel method that uses text representations to direct the perturbation of clean images, resulting in the generation of adversarial images.