How and where does CLIP process negation?

Understand

Various benchmarks have been proposed to test linguistic understanding in pre-trained vision \& language (VL) models.

• Here we build on the existence task from the VALSE benchmark (Parcalabescu et al, 2022) which we use to test models' understanding of negation, a particularly interesting issue for multimodal models.
• However, while such VL benchmarks are useful for measuring model performance, they do not reveal anything about the internal processes through which these models arrive at their outputs in such visio-linguistic tasks.
• We take inspiration from the growing literature on model interpretability to explain the behaviour of VL models on the understanding of negation.