Fetching the paper…

Reading the bibliography…

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation · Around

2024

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

Yuan, Huizhuo, Chen, Zixiang, Ji, Kaixuan et al.

Original
PDF
Post on X

Understand

Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tuning Large Language Models (LLMs).

While cutting-edge diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised fine-tuning, their performance inevitably plateaus after seeing a certain volume of data.
Recently, reinforcement learning (RL) has been employed to fine-tune diffusion models with human preference data, but it requires at least two images ("winner" and "loser" images) for each text prompt.
In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion), where the diffusion model engages in competition with its earlier versions, facilitating an iterative self-improvement process.

Built on

Rank analysis of incomplete block designs: I. the method of paired comparisons
Bradley, R. A · 1952
Earlier work this paper cites.
Temporal difference learning and td-gammon
Tesauro, G · 1995
Earlier work this paper cites.
Integral probability metrics and their generating classes of functions
Müller, A · 1997
Earlier work this paper cites.
Denoising diffusion implicit models
Original
Song, J · 2010
Earlier work this paper cites.
Score-based generative modeling through stochastic differential equations
Original
Song, Y · 2011
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Kingma, D. P · 2013
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J · 2015
Earlier work this paper cites.
Coco-stuff: Thing and stuff classes in context
Caesar, H · 2018
Earlier work this paper cites.
Generative adversarial networks: An overview
Creswell, A · 2018
Earlier work this paper cites.
Generative modeling by estimating gradients of the data distribution
Song, Y · 2019
Earlier work this paper cites.
Denoising diffusion probabilistic models
Ho, J · 2020
Earlier work this paper cites.
Structured denoising diffusion models in discrete state-spaces
Austin, J · 2021
Earlier work this paper cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Nichol, A · 2021
Earlier work this paper cites.
Learning to efficiently sample from diffusion probabilistic models
Original
Watson, D · 2021
Earlier work this paper cites.

Similar

Diffdock: Diffusion steps, twists, and turns for molecular docking
Original
Corso, G · 2022
Cited alongside, same era.
An image is worth one word: Personalizing text-to-image generation using textual inversion
Original
Gal, R · 2022
Cited alongside, same era.
Cascaded diffusion models for high fidelity image generation
Ho, J · 2022
Cited alongside, same era.
Hierarchical text-conditional image generation with clip latents
Original
Ramesh, A · 2022
Cited alongside, same era.
Laion-5b: An open large-scale dataset for training next generation image-text models
Schuhmann, C · 2022
Cited alongside, same era.

Then

Aligning text-to-image models using human feedback
Original
Lee, K · 2023
Later among the works it cites.
Scalable diffusion models with transformers
Peebles, W · 2023
Later among the works it cites.
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Original
Podell, D · 2023
Later among the works it cites.
Aligning text-to-image diffusion models with reward backpropagation
Original
Prabhudesai, M · 2023
Later among the works it cites.
Direct preference optimization: Your language model is secretly a reward model
Original
Rafailov, R · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Scaling autoregressive models for content-rich text-to-image generation

Yu, J · 2022

Cited alongside, same era.

Improving image generation with better captions

Betker, J · 2023

Cited alongside, same era.

Training diffusion models with reinforcement learning

Black, K · 2023

Cited alongside, same era.

Fast sampling via de-randomization for discrete diffusion models

Chen, Z · 2023

Cited alongside, same era.

Directly fine-tuning diffusion models on differentiable rewards

Clark, K · 2023

Cited alongside, same era.

Emu: Enhancing image generation models using photogenic needles in a haystack

Dai, X · 2023

Cited alongside, same era.

Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models

Fan, Y · 2023

Cited alongside, same era.

Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

Ruiz, N · 2023

Later among the works it cites.

A picture is worth a thousand words: Principled recaptioning improves image generation

Segalis, E · 2023

Later among the works it cites.

Diffusion model alignment using direct preference optimization

Wallace, B · 2023

Later among the works it cites.

Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

Wu, X · 2023

Later among the works it cites.

Imagereward: Learning and evaluating human preferences for text-to-image generation

Xu, J · 2023

Later among the works it cites.

Using human feedback to fine-tune diffusion models without any reward model

Yang, K · 2023

Later among the works it cites.

A reparameterized discrete diffusion model for text generation

Zheng, L · 2023

Later among the works it cites.

Self-play fine-tuning converts weak language models to strong language models

Chen, Z · 2024

Closest in time.

Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-image generation

Lee, S. H · 2024

Closest in time.