VideoAgent: Self-Improving Video Generation

Understand

Video generation has been used to generate visual plans for controlling robotic systems.

• Given an image observation and a language instruction, previous work has generated video plans which are then converted to robot controls to be executed.
• However, a major bottleneck in leveraging video generation for control lies in the quality of the generated videos, which often suffer from hallucinatory content and unrealistic physics, resulting in low task success when control actions are extracted from the generated videos.
• While scaling up dataset and model size provides a partial solution, integrating external feedback is both natural and essential for grounding video generation in the real world.