Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding

Understand

As large-scale text-to-image generation models have made remarkable progress in the field of text-to-image generation, many fine-tuning methods have been proposed.

• However, these models often struggle with novel objects, especially with one-shot scenarios.
• Our proposed method aims to address the challenges of generalizability and fidelity in an object-driven way, using only a single input image and the object-specific regions of interest.
• To improve generalizability and mitigate overfitting, in our paradigm, a prototypical embedding is initialized based on the object's appearance and its class, before fine-tuning the diffusion model.