2022

CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Ding, Ming, Zheng, Wendi, Hong, Wenyi et al.

Understand

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images.

In this work, we put forward a solution based on hierarchical transformers and local parallel auto-regressive generation.
We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution.
The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.

Built on

Some methods for classification and analysis of multivariate observations
J. MacQueen et al · 1967
Earlier work this paper cites.
Multiscale structural similarity for image quality assessment
Z. Wang, E. P. Simoncelli, and A. C. Bovik · 2003
Earlier work this paper cites.
Learning a deep convolutional network for image super-resolution
C. Dong, C. C. Loy, K. He, and X. Tang · 2014
Earlier work this paper cites.
Generative adversarial networks
Original
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio · 2014
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Fast generation for convolutional autoregressive models
Original
P. Ramachandran, T. L. Paine, P. Khorrami, M. Babaeizadeh, S. Chang, Y. Zhang, M. A. Hasegawa-Johnson, R. H. Campbell, and T. S. Huang · 2017
Earlier work this paper cites.
Neural discrete representation learning
A. van den Oord, O. Vinyals, and K. Kavukcuoglu · 2017
Earlier work this paper cites.
Attention is all you need
Original
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2018
Earlier work this paper cites.
Subword regularization: Improving neural network translation models with multiple subword candidates
Original
T. Kudo · 2018
Earlier work this paper cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman · 2018
Earlier work this paper cites.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks
T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He · 2018
Earlier work this paper cites.

Similar

The unreasonable effectiveness of deep features as a perceptual metric
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang · 2018
Cited alongside, same era.
Mask-predict: Parallel decoding of conditional masked language models
M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer · 2019
Cited alongside, same era.
The curious case of neural text degeneration
Original
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever · 2019
Cited alongside, same era.
Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis
M. Zhu, P. Pan, W. Chen, and Y. Yang · 2019
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al · 2020
Cited alongside, same era.
Taming transformers for high-resolution image synthesis
Original
P. Esser, R. Rombach, and B. Ommer · 2020
Cited alongside, same era.
Deberta: Decoding-enhanced bert with disentangled attention
Original
P. He, X. Liu, J. Gao, and W. Chen · 2020
Cited alongside, same era.
Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis
Original
M. Tao, H. Tang, S. Wu, N. Sebe, F. Wu, and X.-Y. Jing · 2020
Cited alongside, same era.
Cogview: Mastering text-to-image generation via transformers
M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al · 2021
Cited alongside, same era.
All nlp tasks are generation tasks: A general pretraining framework
Original
Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang · 2021
Cited alongside, same era.

Then

Masked autoencoders are scalable vision learners
Original
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick · 2021
Later among the works it cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen · 2021
Later among the works it cites.
Glancing transformer for non-autoregressive neural machine translation
L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Original
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al · 2021
Later among the works it cites.
Do vision transformers see like convolutional neural networks?
M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy · 2021
Later among the works it cites.
Zero-shot text-to-image generation
Original
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever · 2021
Later among the works it cites.
N \ \backslash " uwa: Visual synthesis pre-training for neural visual world creation
Original
C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan · 2021
Later among the works it cites.
Simmim: A simple framework for masked image modeling
Original
Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu · 2021
Later among the works it cites.
Lafite: Towards language-free training for text-to-image generation
Original
Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun · 2021
Later among the works it cites.
Maskgit: Masked generative image transformer
Original
H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman · 2022
Closest in time.
Make-a-scene: Scene-based text-to-image generation with human priors
Original
O. Gafni, A. Polyak, O. Ashual, S. Sheynin, D. Parikh, and Y. Taigman · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents
Original
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…