Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons

Understand

With Open AI's publishing of their CLIP model (Contrastive Language-Image Pre-training), multi-modal neural networks now provide accessible models that combine reading with visual recognition.

• Their network offers novel ways to probe its dual abilities to read text while classifying visual objects.
• This paper demonstrates several new categories of adversarial attacks, spanning basic typographical, conceptual, and iconographic inputs generated to fool the model into making false or absurd classifications.
• We demonstrate that contradictory text and image signals can confuse the model into choosing false (visual) options.