Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control

Understand

Our goal is for robots to follow natural language instructions like "put the towel next to the microwave." But getting large amounts of labeled data, i.e.

• data that contains demonstrations of tasks labeled with the language instruction, is prohibitive.
• In contrast, obtaining policies that respond to image goals is much easier, because any autonomous trial or demonstration can be labeled in hindsight with its final state as the goal.
• In this work, we contribute a method that taps into joint image- and goal- conditioned policies with language using only a small amount of language data.