Learning to Imitate Object Interactions from Internet Videos

Understand

We study the problem of imitating object interactions from Internet videos.

• This requires understanding the hand-object interactions in 4D, spatially in 3D and over time, which is challenging due to mutual hand-object occlusions.
• In this paper we make two main contributions: (1) a novel reconstruction technique RHOV (Reconstructing Hands and Objects from Videos), which reconstructs 4D trajectories of both the hand and the object using 2D image cues and temporal smoothness constraints; (2) a system for imitating object interactions in a physics simulator with reinforcement learning.
• We apply our reconstruction technique to 100 challenging Internet videos.