SILC: Improving Vision Language Pretraining with Self-Distillation

Understand

Image-Text pretraining on web-scale image caption datasets has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants.

• Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities.
• However, the contrastive objective used by these models only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks.
• In this work, we introduce SILC, a novel framework for vision language pretraining.