Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models

Understand

Diffusion models represent a new paradigm in text-to-image generation.

• Beyond generating high-quality images from text prompts, models such as Stable Diffusion have been successfully extended to the joint generation of semantic segmentation pseudo-masks.
• However, current extensions primarily rely on extracting attentions linked to prompt words used for image synthesis.
• This approach limits the generation of segmentation masks derived from word tokens not contained in the text prompt.