CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

Understand

Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images.

• However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details).
• In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture.
• This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level.