MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion

Understand

Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval.

• However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals.
• Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities.
• We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion.