Rethinking and Red-Teaming Protective Perturbation in Personalized Diffusion Models

Understand

Personalized diffusion models (PDMs) have become prominent for adapting pre-trained text-to-image models to generate images of specific subjects using minimal training data.

• However, PDMs are susceptible to minor adversarial perturbations, leading to significant degradation when fine-tuned on corrupted datasets.
• These vulnerabilities are exploited to create protective perturbations that prevent unauthorized image generation.
• Existing purification methods attempt to red-team the protective perturbation to break the protection but often over-purify images, resulting in information loss.