Vision-Language Pre-Training with Triple Contrastive Learning

Understand

Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss).

• The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text.
• However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations.
• For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by.