Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Understand

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations.

• It is an important step toward reducing laborious human supervision.
• Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations.
• However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation.