Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations

Understand

Most video-and-language representation learning approaches employ contrastive learning, e.g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video pairs.

• However, such learned shared latent spaces are not often optimal, and the modality gap between visual and textual representation can not be fully eliminated.
• In this paper, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations.
• Specifically, we use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases.