Learning Latent Action World Models In The Wild

Understand

Agents capable of reasoning and planning in the real world require the ability of predicting the consequences of their actions.

• While world models possess this capability, they most often require action labels, that can be complex to obtain at scale.
• This motivates the learning of latent action models, that can learn an action space from videos alone.
• Our work addresses the problem of learning latent actions world models on in-the-wild videos, expanding the scope of existing works that focus on simple robotics simulations, video games, or manipulation data.