Learning Invariant Causal Mechanism from Vision-Language Models

Understand

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable success, but its performance can degrade when fine-tuned in out-of-distribution (OOD) scenarios.

• We model the prediction process using a Structural Causal Model (SCM) and show that the causal mechanism involving both invariant and variant factors in training environments differs from that in test environments.
• In contrast, the causal mechanism with solely invariant factors remains consistent across environments.
• We theoretically prove the existence of a linear mapping from CLIP embeddings to invariant factors, which can be estimated using interventional data.