Diffusion Models Need Visual Priors for Image Generation

Understand

Conventional class-guided diffusion models generally succeed in generating images with correct semantic content, but often struggle with texture details.

• This limitation stems from the usage of class priors, which only provide coarse and limited conditional information.
• To address this issue, we propose Diffusion on Diffusion (DoD), an innovative multi-stage generation framework that first extracts visual priors from previously generated samples, then provides rich guidance for the diffusion model leveraging visual priors from the early stages of diffusion sampling.
• Specifically, we introduce a latent embedding module that employs a compression-reconstruction approach to discard redundant detail information from the conditional samples in each stage, retaining only the semantic information for guidance.