Training Diffusion Models with Reinforcement Learning

Understand

Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective.

• However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness.
• In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives.
• We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization (DDPO), that are more effective than alternative reward-weighted likelihood approaches.