Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure

Understand

Our goal in this work is to generate realistic videos given just one initial frame as input.

• Existing unsupervised approaches to this task do not consider the fact that a video typically shows a 3D environment, and that this should remain coherent from frame to frame even as the camera and objects move.
• We address this by developing a model that first estimates the latent 3D structure of the scene, including the segmentation of any moving objects.
• It then predicts future frames by simulating the object and camera dynamics, and rendering the resulting views.