Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations

Understand

Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years.

• Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while overlooking fine-grained cross-modal relationships, e.g., clip-phrase or frame-word.
• In this paper, we propose a novel method, named Hierarchical Cross-Modal Interaction (HCMI), to explore multi-level cross-modal relationships among video-sentence, clip-phrase, and frame-word for text-video retrieval.
• Considering intrinsic semantic frame relations, HCMI performs self-attention to explore frame-level correlations and adaptively cluster correlated frames into clip-level and video-level representations.