H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

Understand

Large-scale pre-training using egocentric human videos has proven effective for robot learning.

• However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots.
• To remedy this, we propose H2R, a human-to-robot data augmentation pipeline that converts egocentric human videos into robot-centric visual data.
• H2R estimates human hand pose from videos, retargets the motion to simulated robotic arms, removes human limbs via segmentation and inpainting, and composites rendered robot embodiments into the original frames with camera-aligned geometry.