Efficient Video Prediction via Sparsely Conditioned Flow Matching

Understand

We introduce a novel generative model for video prediction based on latent flow matching, an efficient alternative to diffusion-based models.

• In contrast to prior work, we keep the high costs of modeling the past during training and inference at bay by conditioning only on a small random set of past frames at each integration step of the image generation process.
• Moreover, to enable the generation of high-resolution videos and to speed up the training, we work in the latent space of a pretrained VQGAN.
• Finally, we propose to approximate the initial condition of the flow ODE with the previous noisy frame.