ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

Understand

Recent Vision-Language Pre-trained (VLP) models based on dual encoder have attracted extensive attention from academia and industry due to their superior performance on various cross-modal tasks and high computational efficiency.

• They attempt to learn cross-modal representation using contrastive learning on image-text pairs, however, the built inter-modal correlations only rely on a single view for each modality.
• Actually, an image or a text contains various potential views, just as humans could capture a real-world scene via diverse descriptions or photos.
• In this paper, we propose ERNIE-ViL 2.0, a Multi-View Contrastive learning framework to build intra-modal and inter-modal correlations between diverse views simultaneously, aiming at learning a more robust cross-modal representation.