Joint Optimization for 4D Human-Scene Reconstruction in the Wild

Understand

Reconstructing human motion and its surrounding environment is crucial for understanding human-scene interaction and predicting human movements in the scene.

• While much progress has been made in capturing human-scene interaction in constrained environments, those prior methods can hardly reconstruct the natural and diverse human motion and scene context from web videos.
• In this work, we propose JOSH, a novel optimization-based method for 4D human-scene reconstruction in the wild from monocular videos.
• JOSH uses techniques in both dense scene reconstruction and human mesh recovery as initialization, and then it leverages the human-scene contact constraints to jointly optimize the scene, the camera poses, and the human motion.