Structured Object-Aware Physics Prediction for Video Modeling and Planning

Understand

When humans observe a physical system, they can easily locate objects, understand their interactions, and anticipate future behavior, even in settings with complicated and previously unseen interactions.

• For computers, however, learning such models from videos in an unsupervised fashion is an unsolved research problem.
• In this paper, we present STOVE, a novel state-space model for videos, which explicitly reasons about objects and their positions, velocities, and interactions.
• It is constructed by combining an image model and a dynamics model in compositional manner and improves on previous work by reusing the dynamics model for inference, accelerating and regularizing training.