Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models

Understand

World models learn the consequences of actions in vision-based interactive systems.

• However, in practical scenarios such as autonomous driving, there commonly exists noncontrollable dynamics independent of the action signals, making it difficult to learn effective world models.
• To tackle this problem, we present a novel reinforcement learning approach named Iso-Dream, which improves the Dream-to-Control framework in two aspects.
• First, by optimizing the inverse dynamics, we encourage the world model to learn controllable and noncontrollable sources of spatiotemporal changes on isolated state transition branches.