Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression

Understand

We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics to generate high-quality data and evaluation in scaling robot learning.

• Building interactive video world models and policies for robotics is difficult due to the challenge of handling diverse settings while maintaining computational efficiency to run in real time.
• HMA uses heterogeneous pre-training from observations and action sequences across different robotic embodiments, domains, and tasks.
• HMA uses masked autoregression to generate quantized or soft tokens for video predictions.