Equivariant Similarity for Vision-Language Foundation Models

Understand

This study explores the concept of equivariance in vision-language foundation models (VLMs), focusing specifically on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks.

• Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes.
• This allows VLMs to generalize better to nuanced and unseen multimodal compositions.
• However, modeling equivariance is challenging as the ground truth of semantic change is difficult to collect.