Object-Centric Latent Action Learning

Understand

Leveraging vast amounts of unlabeled internet video data for embodied AI is currently bottlenecked by the lack of action labels and the presence of action-correlated visual distractors.

• Although recent latent action policy optimization (LAPO) has shown promise in inferring proxy action labels from visual observations, its performance degrades significantly when distractors are present.
• To address this limitation, we propose a novel object-centric latent action learning framework that centers on objects rather than pixels.
• We leverage self-supervised object-centric pretraining to disentangle the movement of the agent and distracting background dynamics.