2022

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Ruiz, Nataniel, Li, Yuanzhen, Jampani, Varun et al.

Understand

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt.

However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts.
In this work, we present a new approach for "personalization" of text-to-image diffusion models.
Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject.

Built on

Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Large scale gan training for high fidelity natural image synthesis
Original
Andrew Brock, Jeff Donahue, and Karen Simonyan · 2018
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson · 2018
Earlier work this paper cites.
St-gan: Spatial transformer generative adversarial networks for image compositing
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey · 2018
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
Photographic text-to-image synthesis with a hierarchically-nested adversarial network
Zizhao Zhang, Yuanpu Xie, and Lin Yang · 2018
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Countering language drift via visual grounding
Jason Lee, Kyunghyun Cho, and Douwe Kiela · 2019
Earlier work this paper cites.
Controllable text-to-image generation
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr · 2019
Earlier work this paper cites.
Object-driven text-to-image synthesis via adversarial training
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao · 2019
Earlier work this paper cites.
Learn, imagine and create: Text-to-image generation from prior knowledge
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao · 2019
Earlier work this paper cites.
Mirrorgan: Learning text-to-image generation by redescription
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao · 2019
Earlier work this paper cites.
Generative modeling by estimating gradients of the data distribution
Yang Song and Stefano Ermon · 2019
Earlier work this paper cites.
Gp-gan: Towards realistic high-resolution image blending
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang · 2019
Earlier work this paper cites.
Dovenet: Deep image harmonization via domain verification
Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang · 2020
Earlier work this paper cites.
Semantic object accuracy for generative text-to-image synthesis
Tobias Hinz, Stefan Heinrich, and Stefan Wermter · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
Analyzing and improving the image quality of stylegan
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila · 2020
Earlier work this paper cites.
Few-shot image generation with elastic weight consolidation
Original
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman · 2020
Earlier work this paper cites.
Countering language drift with seeded iterated learning
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin · 2020
Earlier work this paper cites.
Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Earlier work this paper cites.
Freeze the discriminator: a simple baseline for fine-tuning gans
Sangwoo Mo, Minsu Cho, and Jinwoo Shin · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al · 2020
Earlier work this paper cites.
Few-shot adaptation of generative adversarial networks
Original
Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang · 2020
Earlier work this paper cites.

Similar

Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Cited alongside, same era.
Improved techniques for training score-based generative models
Yang Song and Stefano Ermon · 2020
Cited alongside, same era.
Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis
Original
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao · 2020
Cited alongside, same era.
Minegan: Effective knowledge transfer from gans to target domains with few images
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer · 2020
Cited alongside, same era.
Clip2stylegan: Unsupervised extraction of stylegan edit directions
Original
Rameen Abdal, Peihao Zhu, John Femiani, Niloy J Mitra, and Peter Wonka · 2021
Cited alongside, same era.
Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan · 2021
Cited alongside, same era.
Paint by word, 2021
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba · 2021
Cited alongside, same era.
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Cited alongside, same era.
Instance-conditioned gan
Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, and Adriana Romero Soriano · 2021
Cited alongside, same era.
Ilvr: Conditioning method for denoising diffusion probabilistic models
Original
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon · 2021
Cited alongside, same era.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Cited alongside, same era.
Cogview: Mastering text-to-image generation via transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al · 2021
Cited alongside, same era.

Then

Image super-resolution via iterative refinement
Original
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2021
Later among the works it cites.
Tedigan: Text-guided diverse face image generation and manipulation
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu · 2021
Later among the works it cites.
Blended latent diffusion
Original
Omri Avrahami, Ohad Fried, and Dani Lischinski · 2022
Closest in time.
Blended diffusion for text-driven editing of natural images
Omri Avrahami, Dani Lischinski, and Ohad Fried · 2022
Closest in time.
Text2live: Text-driven layered image and video editing
Original
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel · 2022
Closest in time.
Samurai: Shape and material from unconstrained real-world arbitrary image collections
Original
Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan T Barron, Hendrik Lensch, and Varun Jampani · 2022
Closest in time.
Vqgan-clip: Open domain image generation and editing with natural language guidance
Original
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff · 2022
Closest in time.
Cogview2: Faster and better text-to-image generation via hierarchical transformers
Original
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang · 2022
Closest in time.
Make-a-scene: Scene-based text-to-image generation with human priors
Original
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman · 2022
Closest in time.
An image is worth one word: Personalizing text-to-image generation using textual inversion
Original
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or · 2022
Closest in time.
Prompt-to-prompt image editing with cross attention control
Original
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Closest in time.
Cascaded diffusion models for high fidelity image generation
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans · 2022
Closest in time.
Zero-shot text-guided object generation with dream fields
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole · 2022
Closest in time.
Diffusionclip: Text-guided diffusion models for robust image manipulation
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye · 2022
Closest in time.
Self-distilled stylegan: Towards generation from internet photos
Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel Cohen-Or, and Michal Irani · 2022
Closest in time.
Mystyle: A personalized generative prior
Original
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or · 2022
Closest in time.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
Pivotal tuning for latent-based editing of real images
Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or · 2022
Closest in time.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Closest in time.
Palette: Image-to-image diffusion models
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi · 2022
Closest in time.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi · 2022
Closest in time.
Ref-NeRF: Structured view-dependent appearance for neural radiance fields
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan · 2022
Closest in time.
Scaling autoregressive models for content-rich text-to-image generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…