LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentation

Understand

Zero-shot referring image segmentation aims to locate and segment the target region based on a referring expression, with the primary challenge of aligning and matching semantics across visual and textual modalities without training.

• Previous works address this challenge by utilizing Vision-Language Models and mask proposal networks for region-text matching.
• However, this paradigm may lead to incorrect target localization due to the inherent ambiguity and diversity of free-form referring expressions.
• To alleviate this issue, we present LGD (Leveraging Generative Descriptions), a framework that utilizes the advanced language generation capabilities of Multi-Modal Large Language Models to enhance region-text matching performance in Vision-Language Models.