You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions

Understand

The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera.

• We propose a learning-based approach to estimate the camera wearer's 3D body pose from egocentric video sequences.
• Our key insight is to leverage interactions with another person---whose body pose we can directly observe---as a signal inherently linked to the body pose of the first-person subject.
• We show that since interactions between individuals often induce a well-ordered series of back-and-forth responses, it is possible to learn a temporal model of the interlinked poses even though one party is largely out of view.