Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video

Understand

Creating high-quality and interactive virtual environments, such as games and simulators, often involves complex and costly manual modeling processes.

• In this paper, we present Video2Game, a novel approach that automatically converts videos of real-world scenes into realistic and interactive game environments.
• At the heart of our system are three core components:(i) a neural radiance fields (NeRF) module that effectively captures the geometry and visual appearance of the scene; (ii) a mesh module that distills the knowledge from NeRF for faster rendering; and (iii) a physics module that models the interactions and physical dynamics among the objects.
• By following the carefully designed pipeline, one can construct an interactable and actionable digital replica of the real world.