TextOCVP: Object-Centric Video Prediction with Language Guidance

Understand

Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments.

• Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and predicting future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotics.
• To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions.
• TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames.