Unsupervised Learning for Physical Interaction through Video Prediction

Understand

A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment.

• Many existing methods for learning the dynamics of physical interactions require labeled object information.
• However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical.
• To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames.