TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment

Understand

Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning.

• This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques.
• The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words.
• This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words.