Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

Understand

The success of vision-language models is primarily attributed to effective alignment across modalities such as vision and language.

• However, modality gaps persist in existing alignment algorithms and appear necessary for human perception as evidenced by modality-specific phenomena like visual texture and linguistic tone.
• These observations motivate us to computationally measure and leverage modality gaps to improve downstream tasks.
• We first introduce the Modality Dominance Score (MDS), which attributes multimodal features to specific modalities by categorizing them into three classes: vision-dominant features, language-dominant features, and cross-modal features.