2023

LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation

Cheng, Jiaxin, Liang, Xiao, Shi, Xingjian et al.

Understand

Layout-to-image generation refers to the task of synthesizing photo-realistic images based on semantic layouts.

In this paper, we propose LayoutDiffuse that adapts a foundational diffusion model pretrained on large-scale image or text-image datasets for layout-to-image generation.
By adopting a novel neural adaptor based on layout attention and task-aware prompts, our method trains efficiently, generates images with both high perceptual quality and layout alignment, and needs less data.
Experiments on three datasets show that our method significantly outperforms other 10 generative models based on GANs, VQ-VAE, and diffusion models.

Built on

Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Improved techniques for training gans
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen · 2016
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al · 2017
Earlier work this paper cites.
Neural discrete representation learning
Aaron Van Den Oord, Oriol Vinyals, et al · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Coco-stuff: Thing and stuff classes in context
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari · 2018
Earlier work this paper cites.
Image generation from scene graphs
Justin Johnson, Agrim Gupta, and Li Fei-Fei · 2018
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly · 2019
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Earlier work this paper cites.
Image synthesis from reconfigurable layout and style
Wei Sun and Tianfu Wu · 2019
Earlier work this paper cites.
Image generation from layout
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal · 2019
Earlier work this paper cites.
Yolov4: Optimal speed and accuracy of object detection
Original
Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao · 2020
Earlier work this paper cites.
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2020
Earlier work this paper cites.

Similar

Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
MaskGAN: Towards diverse and interactive facial image manipulation
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo · 2020
Cited alongside, same era.
Maskgan: Towards diverse and interactive facial image manipulation
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo · 2020
Cited alongside, same era.
Denoising diffusion implicit models
Original
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Cited alongside, same era.
Score-based generative modeling through stochastic differential equations
Original
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2020
Cited alongside, same era.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Cited alongside, same era.
Context-aware layout to image generation with enhanced object appearance
Sen He, Wentong Liao, Michael Ying Yang, Yongxin Yang, Yi-Zhe Song, Bodo Rosenhahn, and Tao Xiang · 2021
Cited alongside, same era.
High-resolution complex scene synthesis with transformers
Original
Manuel Jahn, Robin Rombach, and Björn Ommer · 2021
Cited alongside, same era.
Image synthesis from layout with locality-aware mask adaption
Zejian Li, Jingyu Wu, Immanuel Koh, Yongchuan Tang, and Lingyun Sun · 2021
Cited alongside, same era.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Cited alongside, same era.
Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal · 2021
Cited alongside, same era.
Vaes meet diffusion models: Efficient and high-fidelity generation
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar · 2021
Cited alongside, same era.

Then

Tackling the generative learning trilemma with denoising diffusion gans
Original
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat · 2021
Later among the works it cites.
An image is worth one word: Personalizing text-to-image generation using textual inversion
Original
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or · 2022
Later among the works it cites.
Classifier-free diffusion guidance
Original
Jonathan Ho and Tim Salimans · 2022
Later among the works it cites.
Visual prompt tuning
Original
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim · 2022
Later among the works it cites.
Visual prompt tuning
Original
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim · 2022
Later among the works it cites.
Pseudo numerical methods for diffusion models on manifolds
Original
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao · 2022
Later among the works it cites.
Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents
Original
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Later among the works it cites.
Palette: Image-to-image diffusion models
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Later among the works it cites.
Image super-resolution via iterative refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2022
Later among the works it cites.
Make-a-video: Text-to-video generation without text-video data
Original
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al · 2022
Later among the works it cites.
Modeling image composition for complex scene generation
Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, and Dacheng Tao · 2022
Later among the works it cites.
Decoupled multi-task learning with cyclical self-regulation for face parsing
Qingping Zheng, Jiankang Deng, Zheng Zhu, Ying Li, and Stefanos Zafeiriou · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…