2023

Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation

Ma, Yiyang, Yang, Huan, Wang, Wenjing et al.

Understand

Language-guided image generation has achieved great success nowadays by using diffusion models.

However, texts can be less detailed to describe highly-specific subjects such as a particular dog or a certain car, which makes pure text-to-image generation not accurate enough to satisfy user requirements.
In this work, we present a novel Unified Multi-Modal Latent Diffusion (UMM-Diffusion) which takes joint texts and images containing specified subjects as input sequences and generates customized images with the subjects.
To be more specific, both input texts and images are encoded into one unified multi-modal latent space, in which the input images are learned to be projected to pseudo word embedding and can be further combined with text to guide image generation.

Built on

Color transfer between images
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley · 2001
Earlier work this paper cites.
Poisson image editing
Patrick Pérez, Michel Gangnet, and Andrew Blake · 2003
Earlier work this paper cites.
Color harmonization
Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, and Ying-Qing Xu · 2006
Earlier work this paper cites.
Drag-and-drop pasting
Jiaya Jia, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum · 2006
Earlier work this paper cites.
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Multi-scale image harmonization
Kalyan Sunkavalli, Micah K Johnson, Wojciech Matusik, and Hanspeter Pfister · 2010
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Conditional generative adversarial nets
Original
Mehdi Mirza and Simon Osindero · 2014
Earlier work this paper cites.
Fast R-CNN
Ross Girshick · 2015
Earlier work this paper cites.
Generative adversarial text to image synthesis
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee · 2016
Earlier work this paper cites.
Semantic image synthesis via adversarial learning
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo · 2017
Earlier work this paper cites.
Deep image harmonization
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang · 2017
Earlier work this paper cites.
StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2017
Earlier work this paper cites.
ST-GAN: Spatial transformer generative adversarial networks for image compositing
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey · 2018
Earlier work this paper cites.
AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He · 2018
Earlier work this paper cites.
StackGAN++: Realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2018
Earlier work this paper cites.

Similar

Photographic text-to-image synthesis with a hierarchically-nested adversarial network
Zizhao Zhang, Yuanpu Xie, and Lin Yang · 2018
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Cited alongside, same era.
Controllable text-to-image generation
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr · 2019
Cited alongside, same era.
Object-driven text-to-image synthesis via adversarial training
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao · 2019
Cited alongside, same era.
MirrorGAN: Learning text-to-image generation by redescription
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao · 2019
Cited alongside, same era.
GP-GAN: Towards realistic high-resolution image blending
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang · 2019
Cited alongside, same era.
DM-GAN: Dynamic memory generative adversarial networks for text-to-image synthesis
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang · 2019
Cited alongside, same era.
Dovenet: Deep image harmonization via domain verification
Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang · 2020
Cited alongside, same era.
Improving the harmony of the composite image by spatial-separated attention module
Xiaodong Cun and Chi-Man Pun · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Analyzing and improving the image quality of stylegan
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu · 2020
Cited alongside, same era.

Then

Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Later among the works it cites.
NeRF: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2021
Later among the works it cites.
GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Later among the works it cites.
Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal · 2021
Later among the works it cites.
StyleCLIP: Text-driven manipulation of stylegan imagery
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Later among the works it cites.
LAION-400M: Open dataset of clip-filtered 400 million image-text pairs
Original
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki · 2021
Later among the works it cites.
Towards open-world text-guided face image generation and manipulation
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu · 2021
Later among the works it cites.
Training-free structured diffusion guidance for compositional text-to-image synthesis
Original
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang · 2022
Later among the works it cites.
An Image is Worth One Word: Personalizing text-to-image generation using textual inversion
Original
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or · 2022
Later among the works it cites.
Classifier-free diffusion guidance
Original
Jonathan Ho and Tim Salimans · 2022
Later among the works it cites.
AI Illustrator: Translating raw descriptions into images by prompt-based cross-modal generation
Yiyang Ma, Huan Yang, Bei Liu, Jianlong Fu, and Jiaying Liu · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation
Original
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo · 2022
Later among the works it cites.
DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…