2020

Stochastic Latent Residual Video Prediction

Franceschi, Jean-Yves, Delasalles, Edouard, Chen, Mickaël et al.

Understand

Designing video prediction models that account for the inherent uncertainty of the future is challenging.

Most works in the literature are based on stochastic image-autoregressive recurrent networks, which raises several performance and applicability issues.
An alternative is to use fully latent temporal models which untie frame synthesis and temporal dynamics.
However, no such model for stochastic video prediction has been proposed in the literature yet, due to design and training difficulties.

Built on

On information and sufficiency
Kullback, S. and Leibler, R. A · 1951
Earlier work this paper cites.
Neurocomputing: Foundations of Research , chapter Learning Representations by Back-Propagating Errors, pp. 696–699
Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1988
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Incorporating second-order functional knowledge for better option pricing
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R · 2001
Earlier work this paper cites.
Recognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., and Caputo, B · 2004
Earlier work this paper cites.
Latent structured models for human pose estimation
Ionescu, C., Li, F., and Sminchisescu, C · 2011
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Original
Graves, A · 2013
Earlier work this paper cites.
Learning stochastic recurrent networks
Original
Bayer, J. and Osendorfer, C · 2014
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y · 2014
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments
Ionescu, C., Papava, D., Olaru, V., and Sminchisescu, C · 2014
Earlier work this paper cites.
Auto-encoding variational Bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Video (language) modeling: a baseline for generative models of natural videos
Original
Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Collobert, R., and Chopra, S · 2014
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D · 2014
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O · 2014
Earlier work this paper cites.
A recurrent latent variable model for sequential data
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., and Bengio, Y · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Convolutional LSTM network: A machine learning approach for precipitation nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-k., and Woo, W.-c · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Simonyan, K. and Zisserman, A · 2015
Earlier work this paper cites.
Unsupervised learning of video representations using LSTMs
Srivastava, N., Mansimov, E., and Salakhudinov, R · 2015
Earlier work this paper cites.
Dense optical flow prediction from a static image
Walker, J., Gupta, A., and Hebert, M · 2015
Earlier work this paper cites.
Unsupervised learning for physical interaction through video prediction
Finn, C., Goodfellow, I., and Levine, S · 2016
Earlier work this paper cites.
Sequential neural models with stochastic layers
Fraccaro, M., Sønderby, S. K., Paquet, U., and Winther, O · 2016
Earlier work this paper cites.
NIPS 2016 tutorial: Generative adversarial networks
Original
Goodfellow, I · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Dynamic filter networks
Jia, X., De Brabandere, B., Tuytelaars, T., and Van Gool, L · 2016
Earlier work this paper cites.

Similar

Deep multi-scale video prediction beyond mean square error
Mathieu, M., Couprie, C., and LeCun, Y · 2016
Cited alongside, same era.
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., and Chintala, S · 2016
Cited alongside, same era.
Conditional image generation with PixelCNN decoders
van den Oord, A., Kalchbrenner, N., Espeholt, L., Kavukcuoglu, K., Vinyals, O., and Graves, A · 2016
Cited alongside, same era.
An uncertain future: Forecasting from static images using variational autoencoders
Walker, J., Doersch, C., Gupta, A., and Hebert, M · 2016
Cited alongside, same era.
Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks
Xue, T., Wu, J., Bouman, K. L., and Freeman, W. T · 2016
Cited alongside, same era.
Real-time video super-resolution with spatio-temporal networks and motion compensation
Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., and Shi, W · 2017
Cited alongside, same era.
Unsupervised learning of disentangled representations from video
Denton, E. and Birodkar, V · 2017
Cited alongside, same era.
Self-supervised visual planning with temporal skip connections
Ebert, F., Finn, C., Lee, A. X., and Levine, S · 2017
Cited alongside, same era.
A disentangled recognition and nonlinear dynamics model for unsupervised learning
Fraccaro, M., Kamronn, S., Paquet, U., and Winther, O · 2017
Cited alongside, same era.
β \beta -VAE: Learning basic visual concepts with a constrained variational framework
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A · 2017
Cited alongside, same era.
Video pixel networks
Kalchbrenner, N., van den Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K · 2017
Cited alongside, same era.
Deep variational Bayes filters: Unsupervised learning of state space models from raw data
Karl, M., Soelch, M., Bayer, J., and van der Smagt, P · 2017
Cited alongside, same era.

Then

Mixed precision training
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H · 2018
Later among the works it cites.
Black-box variational inference for stochastic differential equations
Ryder, T., Golightly, A., McGough, A. S., and Prangle, D · 2018
Later among the works it cites.
MoCoGAN: Decomposing motion and content for video generation
Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J · 2018
Later among the works it cites.
Towards accurate generative models of video: A new metric & challenges
Original
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S · 2018
Later among the works it cites.
Relational neural expectation maximization: Unsupervised discovery of objects and their interactions
van Steenkiste, S., Chang, M., Greff, K., and Schmidhuber, J · 2018
Later among the works it cites.
Video-to-video synthesis
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., and Catanzaro, B · 2018
Later among the works it cites.
Hierarchical long-term video prediction without supervision
Wichers, N., Villegas, R., Erhan, D., and Lee, H · 2018
Later among the works it cites.
Disentangled sequential autoencoder
Yingzhen, L. and Mandt, S · 2018
Later among the works it cites.
Invertible residual networks
Behrmann, J., Grathwohl, W., Chen, R. T. Q., Duvenaud, D., and Jacobsen, J.-H · 2019
Later among the works it cites.
Improved conditional VRNNs for video prediction
Castrejon, L., Ballas, N., and Courville, A · 2019
Later among the works it cites.
GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
De Brouwer, E., Simm, J., Arany, A., and Moreau, Y · 2019
Later among the works it cites.
Optimal unsupervised domain translation
de Bézenac, E., Ayed, I., and Gallinari, P · 2019
Later among the works it cites.
Cubic LSTMs for video prediction
Fan, H., Zhu, L., and Yang, Y · 2019
Later among the works it cites.
Disentangling propagation and generation for video prediction
Gao, H., Xu, H., Cai, Q.-Z., Wang, R., Yu, F., and Darrell, T · 2019
Later among the works it cites.
Temporal difference variational auto-encoder
Gregor, K., Papamakarios, G., Besse, F., Buesing, L., and Weber, T · 2019
Later among the works it cites.
Learning latent dynamics for planning from pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J · 2019
Later among the works it cites.
Modeling parts, structure, and system dynamics via predictive learning
Liu, Z., Wu, J., Xu, Z., Sun, C., Murphy, K., Freeman, W. T., and Tenenbaum, J. B · 2019
Later among the works it cites.
Unsupervised learning of object structure and dynamics from videos
Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., and Lee, H · 2019
Later among the works it cites.
PyTorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S · 2019
Later among the works it cites.
Residual networks as flows of diffeomorphisms
Rousseau, F., Drumetz, L., and Fablet, R · 2019
Later among the works it cites.
Latent ordinary differential equations for irregularly-sampled time series
Rubanova, Y., Chen, R. T. Q., and Duvenaud, D · 2019
Later among the works it cites.
High fidelity video prediction with large stochastic recurrent neural networks
Villegas, R., Pathak, A., Kannan, H., Erhan, D., Le, Q. V., and Lee, H · 2019
Later among the works it cites.
ODE 2 VAE: Deep generative second order odes with Bayesian neural networks
Yıldız, C., Heinonen, M., and Lahdesmaki, H · 2019
Later among the works it cites.
Exploring spatial-temporal multi-frequency analysis for high-fidelity and temporal-consistency video prediction
Jin, B., Hu, Y., Tang, Q., Niu, J., Shi, Z., Han, Y., and Li, X · 2020
Closest in time.
VideoFlow: A conditional flow-based model for stochastic video generation
Kumar, M., Babaeizadeh, M., Erhan, D., Finn, C., Levine, S., Dinh, L., and Kingma, D · 2020
Closest in time.
Disentangling physical dynamics from unknown factors for unsupervised video prediction
Le Guen, V. and Thome, N · 2020
Closest in time.
Scaling autoregressive video models
Weissenborn, D., Täckström, O., and Uszkoreit, J · 2020
Closest in time.
Future video synthesis with object motion prediction
Wu, Y., Gao, R., Park, J., and Chen, Q · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…