A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

Understand

Aligning text-to-image diffusion model (T2I) with preference has been gaining increasing research attention.

• While prior works exist on directly optimizing T2I by preference data, these methods are developed under the bandit assumption of a latent reward on the entire diffusion reverse chain, while ignoring the sequential nature of the generation process.
• This may harm the efficacy and efficiency of preference alignment.
• In this paper, we take on a finer dense reward perspective and derive a tractable alignment objective that emphasizes the initial steps of the T2I reverse chain.