CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally

Understand

CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks.

• However, recent studies have questioned its ability to represent compositional concepts effectively.
• These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships.
• In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text.