Environment Predictive Coding for Embodied Agents

Understand

We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents.

• In contrast to prior work on self-supervised learning for images, we aim to jointly encode a series of images gathered by an agent as it moves about in 3D environments.
• We learn these representations via a zone prediction task, where we intelligently mask out portions of an agent's trajectory and predict them from the unmasked portions, conditioned on the agent's camera poses.
• By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigation-oriented tasks.