2021

N\"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Wu, Chenfei, Liang, Jian, Ji, Lei et al.

Understand

This paper presents a unified multimodal pre-trained model called N\"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks.

To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively.
A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity.
We evaluate N\"UWA on 8 downstream tasks.

Built on

Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick · 2014
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Original
Alec Radford, Luke Metz, and Soumith Chintala · 2015
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Original
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Earlier work this paper cites.
Unsupervised learning for physical interaction through video prediction
Chelsea Finn, Ian Goodfellow, and Sergey Levine · 2016
Earlier work this paper cites.
Improved techniques for training gans
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen · 2016
Earlier work this paper cites.
Conditional image generation with pixelcnn decoders
Original
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Pixel recurrent neural networks
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Msr-vtt: A large video description dataset for bridging video and language
Jun Xu, Tao Mei, Ting Yao, and Yong Rui · 2016
Earlier work this paper cites.
Stochastic variational video prediction
Original
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine · 2017
Earlier work this paper cites.
Self-Supervised Visual Planning with Temporal Skip Connections
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
The kinetics human action video dataset
Original
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, and Paul Natsev · 2017
Earlier work this paper cites.
Neural discrete representation learning
Original
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Stochastic video generation with a learned prior
Emily Denton and Rob Fergus · 2018
Earlier work this paper cites.
Stochastic adversarial video prediction
Original
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine · 2018
Earlier work this paper cites.
Video generation from text
Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin · 2018
Earlier work this paper cites.

Similar

Image transformer
Original
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, \Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran · 2018
Cited alongside, same era.
Mocogan: Decomposing motion and content for video generation
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz · 2018
Cited alongside, same era.
Towards accurate generative models of video: A new metric & challenges
Original
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly · 2018
Cited alongside, same era.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He · 2018
Cited alongside, same era.
Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf · 2019
Cited alongside, same era.
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Original
Andrew Brock, Jeff Donahue, and Karen Simonyan · 2019
Cited alongside, same era.
Generating long sequences with sparse transformers
Original
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 2019
Cited alongside, same era.
What does bert look at? an analysis of bert’s attention
Original
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning · 2019
Cited alongside, same era.
Axial Attention in Multidimensional Transformers
Original
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans · 2019
Cited alongside, same era.
Videoflow: A conditional flow-based model for stochastic video generation
Original
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma · 2019
Cited alongside, same era.
Moments in time dataset: One million videos for event understanding
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, and Carl Vondrick · 2019
Cited alongside, same era.
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Cited alongside, same era.

Then

The open images dataset v4
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, and Alexander Kolesnikov · 2020
Later among the works it cites.
Transformation-based adversarial video prediction on large-scale data
Original
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan · 2020
Later among the works it cites.
Latent Video Transformer
Original
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev · 2020
Later among the works it cites.
Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis
Original
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao · 2020
Later among the works it cites.
Scaling autoregressive video models
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit · 2020
Later among the works it cites.
Object-contextual representations for semantic segmentation
Yuhui Yuan, Xilin Chen, and Jingdong Wang · 2020
Later among the works it cites.
VideoGen: Generative Modeling of Videos using VQ-VAE and Transformers
Yunzhi Zhang, Wilson Yan, Pieter Abbeel, and Aravind Srinivas · 2020
Later among the works it cites.
Paint by word
Original
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba · 2021
Closest in time.
CogView: Mastering Text-to-Image Generation via Transformers
Original
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang · 2021
Closest in time.
Taming Transformers for High-Resolution Image Synthesis
Original
Patrick Esser, Robin Rombach, and Björn Ommer · 2021
Closest in time.
VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild
Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang · 2021
Closest in time.
CCVS: Context-aware Controllable Video Synthesis
Original
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid · 2021
Closest in time.
Learning Transferable Visual Models From Natural Language Supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Closest in time.
Zero-Shot Text-to-Image Generation
Original
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Closest in time.
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
Original
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan · 2021
Closest in time.
VideoGPT: Video Generation using VQ-VAE and Transformers
Original
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas · 2021
Closest in time.
Cross-modal contrastive learning for text-to-image generation
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

N\&#34;UWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Understand

Built on

Similar

Then

Beyond the bibliography

Built on

Similar

Then

Beyond the bibliography

N\"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion