2022

Diffusion Models for Video Prediction and Infilling

Höppe, Tobias, Mehrjou, Arash, Bauer, Stefan et al.

Understand

Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions.

This requires strong, temporally coherent generative capabilities.
Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain.
We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training.

Built on

Efficient video generation on complex datasets
Original
Aidan Clark, Jeff Donahue, and Karen Simonyan · 1907
Earlier work this paper cites.
Estimation of non-normalized statistical models by score matching
Aapo Hyvärinen and Peter Dayan · 2005
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
Pascal Vincent · 2011
Earlier work this paper cites.
Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah · 2012
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Dense optical flow prediction from a static image
Jacob Walker, Abhinav Gupta, and Martial Hebert · 2015
Earlier work this paper cites.
Improved Techniques for Training GANs
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen · 2016
Earlier work this paper cites.
Generating videos with scene dynamics
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba · 2016
Earlier work this paper cites.
Self-supervised visual planning with temporal skip connections
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine · 2017
Earlier work this paper cites.
Deep visual foresight for planning robot motion
Chelsea Finn and Sergey Levine · 2017
Earlier work this paper cites.
Generating the future with adversarial transformers
Carl Vondrick and Antonio Torralba · 2017
Earlier work this paper cites.
Visual forecasting by imitating dynamics in natural sequences
Kuo-Hao Zeng, William B. Shen, De-An Huang, Min Sun, and Juan Carlos Niebles · 2017
Earlier work this paper cites.
Towards high resolution video generation with progressive growing of sliced wasserstein gans, 2018
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool · 2018
Earlier work this paper cites.
Stochastic variational video prediction
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine · 2018
Earlier work this paper cites.
A short note about kinetics-600, 2018
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman · 2018
Earlier work this paper cites.
Stochastic video generation with a learned prior
Emily Denton and Rob Fergus · 2018
Earlier work this paper cites.
Super slomo: High quality estimation of multiple intermediate frames for video interpolation
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz · 2018
Earlier work this paper cites.
Human action recognition and prediction: A survey, 2018
Yu Kong and Yun Fu · 2018
Earlier work this paper cites.
Stochastic adversarial video prediction, 2018
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine · 2018
Earlier work this paper cites.

Similar

Deep energy estimator networks
Original
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen · 2018
Cited alongside, same era.
Mocogan: Decomposing motion and content for video generation
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz · 2018
Cited alongside, same era.
Towards accurate generative models of video: A new metric & challenges
Original
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly · 2018
Cited alongside, same era.
Stochastic dynamics for video infilling, 2018
Qiangeng Xu, Hanwang Zhang, Weiyue Wang, Peter N. Belhumeur, and Ulrich Neumann · 2018
Cited alongside, same era.
Learning latent dynamics for planning from pixels
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson · 2019
Cited alongside, same era.
Stochastic video generation with disentangled representations
Maomao Li, Chun Yuan, Zhihui Lin, Zhuobin Zheng, and Yangyang Cheng · 2019
Cited alongside, same era.
Predicting future instance segmentation with contextual pyramid convlstms
Jiangxin Sun, Jiafeng Xie, Jian-Fang Hu, Zihang Lin, Jianhuang Lai, Wenjun Zeng, and Wei-shi Zheng · 2019
Cited alongside, same era.
Recurrent flow-guided semantic forecasting
Adam Terwilliger, Garrick Brazil, and Xiaoming Liu · 2019
Cited alongside, same era.
Learning gradient fields for shape generation, 2020
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Probabilistic future prediction for video scene understanding, 2020
Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall · 2020
Cited alongside, same era.
Lower dimensional kernels for video discriminators
Emmanuel Kahembwe and Subramanian Ramamoorthy · 2020
Cited alongside, same era.

Then

Diffusion-based representation learning
Original
Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou · 2021
Later among the works it cites.
Fitvid: Overfitting in pixel-level video prediction
Original
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan · 2021
Later among the works it cites.
Conditional image generation with score-based diffusion models, 2021
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann · 2021
Later among the works it cites.
Wavegrad: Estimating gradients for waveform generation
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan · 2021
Later among the works it cites.
Diffusion models beat GANs on image synthesis
Prafulla Dhariwal and Alexander Quinn Nichol · 2021
Later among the works it cites.
Score-based generative modeling with critically-damped langevin diffusion
Original
Tim Dockhorn, Arash Vahdat, and Karsten Kreis · 2021
Later among the works it cites.
CCVS: Context-aware controllable video synthesis
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid · 2021
Later among the works it cites.
Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal · 2021
Later among the works it cites.
Clockwork variational autoencoders
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner · 2021
Later among the works it cites.
Score-based generative modeling through stochastic differential equations, 2021
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2021
Later among the works it cites.
CSDI: Conditional score-based diffusion models for probabilistic time series imputation
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon · 2021
Later among the works it cites.
Greedy hierarchical variational autoencoders for large-scale video prediction
Bohan Wu, Suraj Nair, Roberto Martín-Martín, Li Fei-Fei, and Chelsea Finn · 2021
Later among the works it cites.
Videogpt: Video generation using vq-vae and transformers, 2021
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas · 2021
Later among the works it cites.
Flexible diffusion modeling of long videos, 2022
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood · 2022
Closest in time.
Video diffusion models, 2022
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet · 2022
Closest in time.
Repaint: Inpainting using denoising diffusion probabilistic models, 2022
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool · 2022
Closest in time.
From points to functions: Infinite-dimensional representations in diffusion models
Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, and Arash Mehrjou · 2022
Closest in time.
Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022
Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal · 2022
Closest in time.
Diffusion probabilistic modeling for video generation, 2022
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…