Diffusion models as plug-and-play priors

Understand

We consider the problem of inferring high-dimensional data $\mathbf{x}$ in a model that consists of a prior $p(\mathbf{x})$ and an auxiliary differentiable constraint $c(\mathbf{x},\mathbf{y})$ on $x$ given some additional information $\mathbf{y}$.

• In this paper, the prior is an independently trained denoising diffusion generative model.
• The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources.
• The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation.