Distilling Diversity and Control in Diffusion Models

Understand

Distilled diffusion models generate images in far fewer timesteps but suffer from reduced sample diversity when generating multiple outputs from the same prompt.

• To understand this phenomenon, we first investigate whether distillation damages concept representations by examining if the required diversity is properly learned.
• Surprisingly, distilled models retain the base model's representational structure: control mechanisms like Concept Sliders and LoRAs transfer seamlessly without retraining, and SliderSpace analysis reveals distilled models possess variational directions needed for diversity yet fail to activate them.
• This redirects our investigation to understanding how the generation dynamics differ between base and distilled models.