DisCLIP: Open-Vocabulary Referring Expression Generation

Understand

Referring Expressions Generation (REG) aims to produce textual descriptions that unambiguously identifies specific objects within a visual scene.

• Traditionally, this has been achieved through supervised learning methods, which perform well on specific data distributions but often struggle to generalize to new images and concepts.
• To address this issue, we present a novel approach for REG, named DisCLIP, short for discriminative CLIP.
• We build on CLIP, a large-scale visual-semantic model, to guide an LLM to generate a contextual description of a target concept in an image while avoiding other distracting concepts.