DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models

Understand

Collecting and annotating images with pixel-wise labels is time-consuming and laborious.

• In contrast, synthetic data can be freely available using a generative model (e.g., DALL-E, Stable Diffusion).
• In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training.
• Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image, which is natural and seamless to extend the text-driven image synthesis to semantic mask generation.