Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals

Understand

Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments.

• While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied.
• In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric.
• We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference.