TokenFlow: Rethinking Fine-grained Cross-modal Alignment in Vision-Language Retrieval

Understand

Most existing methods in vision-language retrieval match two modalities by either comparing their global feature vectors which misses sufficient information and lacks interpretability, detecting objects in images or videos and aligning the text with fine-grained features which relies on complicated model designs, or modeling fine-grained interaction via cross-attention upon visual and textual tokens which suffers from inferior efficiency.

• To address these limitations, some recent works simply aggregate the token-wise similarities to achieve fine-grained alignment, but they lack intuitive explanations as well as neglect the relationships between token-level features and global representations with high-level semantics.
• In this work, we rethink fine-grained cross-modal alignment and devise a new model-agnostic formulation for it.
• We additionally demystify the recent popular works and subsume them into our scheme.