Modeling Caption Diversity in Contrastive Vision-Language Pretraining

Understand

There are a thousand ways to caption an image.

• Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an image.
• In this work, we introduce Llip, Latent Language Image Pretraining, which models the diversity of captions that could match an image.
• Llip's vision encoder outputs a set of visual features that are mixed into a final representation by conditioning on information derived from the text.