Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

Understand

The field of visual representation learning has seen explosive growth in the past years, but its benefits in robotics have been surprisingly limited so far.

• Prior work uses generic visual representations as a basis to learn (task-specific) robot action policies (e.g., via behavior cloning).
• While the visual representations do accelerate learning, they are primarily used to encode visual observations.
• Thus, action information has to be derived purely from robot data, which is expensive to collect! In this work, we present a scalable alternative where the visual representations can help directly infer robot actions.