2023

Scaling up GANs for Text-to-Image Synthesis

Kang, Minguk, Zhu, Jun-Yan, Zhang, Richard et al.

Understand

The recent success of text-to-image synthesis has taken the world by storm and captured the general public's imagination.

From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models.
GANs used to be the de facto choice, with techniques like StyleGAN.
With DALL-E 2, auto-regressive and diffusion models became the new standard for large-scale generative models overnight.

Built on

Filters for common resampling tasks
Ken Turkowski · 1990
Earlier work this paper cites.
A text-to-picture synthesis system for augmenting communication
Xiaojin Zhu, Andrew B Goldberg, Mohamed Eldawy, Charles R Dyer, and Bradley Strock · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Caltech-UCSD Birds 200
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona · 2010
Earlier work this paper cites.
Generative Adversarial Nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2015
Earlier work this paper cites.
Deep generative image models using a laplacian pyramid of adversarial networks
Emily L Denton, Soumith Chintala, Rob Fergus, et al · 2015
Earlier work this paper cites.
Image super-resolution using deep convolutional networks
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Original
Alec Radford, Luke Metz, and Soumith Chintala · 2015
Earlier work this paper cites.
Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
Original
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao · 2015
Earlier work this paper cites.
Dynamic filter networks
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool · 2016
Earlier work this paper cites.
Generating Images from Captions with Attention
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov · 2016
Earlier work this paper cites.
Generative adversarial text to image synthesis
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee · 2016
Earlier work this paper cites.
Learning what and where to draw
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee · 2016
Earlier work this paper cites.
Improved Techniques for Training GANs
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen · 2016
Earlier work this paper cites.
Generative visual manipulation on the natural image manifold
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros · 2016
Earlier work this paper cites.
Wasserstein Generative Adversarial Networks
Martin Arjovsky, Soumith Chintala, and Léon Bottou · 2017
Earlier work this paper cites.
Neural Photo Editing with Introspective Adversarial Networks
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston · 2017
Earlier work this paper cites.
Improved training of wasserstein gans
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville · 2017
Earlier work this paper cites.
Hypernetworks
David Ha, Andrew Dai, and Quoc V Le · 2017
Earlier work this paper cites.
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Earlier work this paper cites.
Photo-realistic single image super-resolution using a generative adversarial network
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al · 2017
Earlier work this paper cites.
Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2017
Earlier work this paper cites.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros · 2017
Earlier work this paper cites.
Multimodal unsupervised image-to-image translation
Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz · 2018
Earlier work this paper cites.
Progressive growing of gans for improved quality, stability, and variation
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2018
Earlier work this paper cites.
Diverse image-to-image translation via disentangled representations
Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang · 2018
Earlier work this paper cites.
Which training methods for gans do actually converge?
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin · 2018
Earlier work this paper cites.
Which Training Methods for GANs do actually Converge?
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger · 2018
Earlier work this paper cites.
cGANs with Projection Discriminator
Takeru Miyato and Masanori Koyama · 2018
Earlier work this paper cites.
Image manipulation with perceptual discriminators
Diana Sungatullina, Egor Zakharov, Dmitry Ulyanov, and Victor Lempitsky · 2018
Earlier work this paper cites.
Esrgan: Enhanced super-resolution generative adversarial networks
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy · 2018
Earlier work this paper cites.
Pay Less Attention with Lightweight and Dynamic Convolutions
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli · 2018
Earlier work this paper cites.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He · 2018
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
Image2stylegan: How to embed images into the stylegan latent space?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2019
Earlier work this paper cites.

Similar

Large Scale GAN Training for High Fidelity Natural Image Synthesis
Andrew Brock, Jeff Donahue, and Karen Simonyan · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Cited alongside, same era.
Improved Precision and Recall Metric for Assessing Generative Models
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila · 2019
Cited alongside, same era.
Decoupled Weight Decay Regularization
Ilya Loshchilov and Frank Hutter · 2019
Cited alongside, same era.
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Cited alongside, same era.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 2019
Cited alongside, same era.
Self-Attention Generative Adversarial Networks
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena · 2019
Cited alongside, same era.
Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang · 2019
Cited alongside, same era.
Densely residual laplacian super-resolution
Saeed Anwar and Nick Barnes · 2020
Cited alongside, same era.
Generative pretraining from pixels
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever · 2020
Cited alongside, same era.
Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, and P. Abbeel · 2020
Cited alongside, same era.
GANSpace: Discovering Interpretable GAN Controls
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris · 2020
Cited alongside, same era.

Then

Cross-Modal Contrastive Learning for Text-to-Image Generation
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang · 2021
Later among the works it cites.
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu · 2021
Later among the works it cites.
ediffi: Text-to-image diffusion models with an ensemble of expert denoisers
Original
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al · 2022
Later among the works it cites.
COYO-700M: Image-Text Pair Dataset
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim · 2022
Later among the works it cites.
Any-resolution training for high-resolution image synthesis
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang · 2022
Later among the works it cites.
Maskgit: Masked generative image transformer
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman · 2022
Later among the works it cites.
Cogview2: Faster and better text-to-image generation via hierarchical transformers
Original
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang · 2022
Later among the works it cites.
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman · 2022
Later among the works it cites.
Cascaded Diffusion Models for High Fidelity Image Generation
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans · 2022
Later among the works it cites.
Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans · 2022
Later among the works it cites.
StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis
Original
Minguk Kang, Joonghyuk Shin, and Jaesik Park · 2022
Later among the works it cites.
Ensembling off-the-shelf models for gan training
Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu · 2022
Later among the works it cites.
The Role of ImageNet Classes in Fr
Original
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen · 2022
Later among the works it cites.
Autoregressive Image Generation using Residual Quantization
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han · 2022
Later among the works it cites.
ViTGAN: Training GANs with vision transformers
Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu · 2022
Later among the works it cites.
Efficient spatially sparse inference for conditional gans and diffusion models
Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun-Yan Zhu · 2022
Later among the works it cites.
Pseudo Numerical Methods for Diffusion Models on Manifolds
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao · 2022
Later among the works it cites.
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps
Original
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu · 2022
Later among the works it cites.
On distillation of guided diffusion models
Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans · 2022
Later among the works it cites.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2022
Later among the works it cites.
DALL·E API
OpenAI · 2022
Later among the works it cites.
On Aliased Resizing and Surprising Subtleties in GAN Evaluation
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu · 2022
Later among the works it cites.
Scalable Diffusion Models with Transformers
Original
William Peebles and Saining Xie · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
Stable Diffusion
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Later among the works it cites.
Image super-resolution via iterative refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2022
Later among the works it cites.
Progressive distillation for fast sampling of diffusion models
Tim Salimans and Jonathan Ho · 2022
Later among the works it cites.
Stylegan-xl: Scaling stylegan to large diverse datasets
Axel Sauer, Katja Schwarz, and Andreas Geiger · 2022
Later among the works it cites.
LAION-5B: An open large-scale dataset for training next generation image-text models
Original
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al · 2022
Later among the works it cites.
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu · 2022
Later among the works it cites.
Scaling autoregressive models for content-rich text-to-image generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al · 2022
Later among the works it cites.
Lafite: Towards language-free training for text-to-image generation
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun · 2022
Later among the works it cites.
Muse: Text-to-image generation via masked generative transformers
Original
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al · 2023
Closest in time.
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Original
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila · 2023
Closest in time.
GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
Original
Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…