Dissecting Representation Misalignment in Contrastive Learning via Influence Function

Understand

Contrastive learning, commonly applied in large-scale multimodal models, often relies on data from diverse and often unreliable sources, which can include misaligned or mislabeled text-image pairs.

• This frequently leads to robustness issues and hallucinations, ultimately causing performance degradation.
• Data valuation is an efficient way to detect and trace these misalignments.
• Nevertheless, existing methods are computationally expensive for large-scale models.