2022

Imagen Video: High Definition Video Generation with Diffusion Models

Ho, Jonathan, Chan, William, Saharia, Chitwan et al.

Understand

We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models.

Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models.
We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models.
In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting.

Built on

Video (language) modeling: a baseline for generative models of natural videos
Original
Marc’Aurelio Ranzato, Arthur D. Szlam, Joan Bruna, Michaël Mathieu, Ronan Collobert, and Sumit Chopra · 2014
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox · 2015
Earlier work this paper cites.
Convolutional lstm network: A machine learning approach for precipitation nowcasting
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang chun Woo · 2015
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Unsupervised learning for physical interaction through video prediction
Original
Chelsea Finn, Ian J. Goodfellow, and Sergey Levine · 2016
Earlier work this paper cites.
Deep multi-scale video prediction beyond mean square error
Original
Michaël Mathieu, Camille Couprie, and Yann LeCun · 2016
Earlier work this paper cites.
Generating videos with scene dynamics
Original
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba · 2016
Earlier work this paper cites.
Identifying and Reducing Gender Bias in Word-Level Language Models
Shikha Bordia and Samuel R. Bowman · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Original
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Video pixel networks
Original
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu · 2017
Earlier work this paper cites.
PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma · 2017
Earlier work this paper cites.
Stochastic variational video prediction
Original
Mohammad Babaeizadeh, Chelsea Finn, D. Erhan, Roy H. Campbell, and Sergey Levine · 2018
Earlier work this paper cites.
Generative Modeling by Estimating Gradients of the Data Distribution
Yang Song and Stefano Ermon · 2019
Earlier work this paper cites.
Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
Original
Belinda Tzen and Maxim Raginsky · 2019
Earlier work this paper cites.
FVD: A new Metric for Video Generation
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly · 2019
Earlier work this paper cites.

Similar

Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Videoflow: A conditional flow-based model for stochastic video generation
Manoj Kumar, Mohammad Babaeizadeh, D. Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma · 2020
Cited alongside, same era.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Cited alongside, same era.
Denoising diffusion implicit models
Original
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Cited alongside, same era.
Fitvid: Overfitting in pixel-level video prediction, 2021
Original
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan · 2021
Cited alongside, same era.
On the dangers of stochastic parrots: Can language models be too big?
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell · 2021
Cited alongside, same era.
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Original
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe · 2021
Cited alongside, same era.
Cogview: Mastering text-to-image generation via transformers, 2021
Original
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang · 2021
Cited alongside, same era.
Clipscore: A reference-free evaluation metric for image captioning
Original
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi · 2021
Cited alongside, same era.
Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans · 2021
Cited alongside, same era.
Variational diffusion models
Original
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho · 2021
Cited alongside, same era.
DiffWave: A Versatile Diffusion Model for Audio Synthesis
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro · 2021
Cited alongside, same era.

Then

Benchmark for compositional text-to-image synthesis
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach · 2021
Later among the works it cites.
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2021
Later among the works it cites.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alex Nichol · 2022
Closest in time.
Maskvit: Masked visual pre-training for video prediction
Original
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Mart’in-Mart’in, and Li Fei-Fei · 2022
Closest in time.
Flexible diffusion modeling of long videos
Original
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood · 2022
Closest in time.
Elucidating the design space of diffusion-based generative models
Original
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine · 2022
Closest in time.
On distillation of guided diffusion models
Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans · 2022
Closest in time.
Hierarchical Text-Conditional Image Generation with CLIP Latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Closest in time.
Progressive Distillation for Fast Sampling of Diffusion Models
Tim Salimans and Jonathan Ho · 2022
Closest in time.
Make-A-Video: Text-to-Video Generation without Text-Video Data
Original
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman · 2022
Closest in time.
Deblurring via Stochastic Refinement
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar · 2022
Closest in time.
Diffusion Probabilistic Modeling for Video Generation
Original
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt · 2022
Closest in time.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, and Yonghui Wu Jason Baldridge · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…