Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation

Understand

Pre-trained vision-language models, e.g., CLIP, have been successfully applied to zero-shot semantic segmentation.

• Existing CLIP-based approaches primarily utilize visual features from the last layer to align with text embeddings, while they neglect the crucial information in intermediate layers that contain rich object details.
• However, we find that directly aggregating the multi-level visual features weakens the zero-shot ability for novel classes.
• The large differences between the visual features from different layers make these features hard to align well with the text embeddings.