COBRA: Contrastive Bi-Modal Representation Algorithm

Understand

There are a wide range of applications that involve multi-modal data, such as cross-modal retrieval, visual question-answering, and image captioning.

• Such applications are primarily dependent on aligned distributions of the different constituent modalities.
• Existing approaches generate latent embeddings for each modality in a joint fashion by representing them in a common manifold.
• However these joint embedding spaces fail to sufficiently reduce the modality gap, which affects the performance in downstream tasks.