Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation

Understand

Recent advancements in pre-trained vision-language models like CLIP have enabled the task of open-vocabulary segmentation.

• CLIP demonstrates impressive zero-shot capabilities in various downstream tasks that require holistic image understanding.
• However, due to the image-level contrastive learning and fully global feature interaction, ViT-based CLIP struggles to capture local details, resulting in poor performance in segmentation tasks.
• Our analysis of ViT-based CLIP reveals that anomaly tokens emerge during the forward process, attracting disproportionate attention from normal patch tokens and thereby diminishing spatial awareness.