ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flow

Understand

One of the central challenges preventing robots from acquiring complex manipulation skills is the prohibitive cost of collecting large-scale robot demonstrations.

• In contrast, humans are able to learn efficiently by watching others interact with their environment.
• To bridge this gap, we introduce semantic action flow as a core intermediate representation capturing the essential spatio-temporal manipulator-object interactions, invariant to superficial visual differences.
• We present ViSA-Flow, a framework that learns this representation self-supervised from unlabeled large-scale video data.