Sketch-Guided Text-to-Image Diffusion Models

Understand

Text-to-Image models have introduced a remarkable leap in the evolution of machine learning, demonstrating high-quality synthesis of images from a given text-prompt.

• However, these powerful pretrained models still lack control handles that can guide spatial properties of the synthesized images.
• In this work, we introduce a universal approach to guide a pretrained text-to-image diffusion model, with a spatial map from another domain (e.g., sketch) during inference time.
• Unlike previous works, our method does not require to train a dedicated model or a specialized encoder for the task.