Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting

Understand

Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions.

• However, existing works still face a main challenge, concept overfitting.
• To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities, i.e, backgrounds, layouts, styles.
• To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively.