CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models

Understand

Generalist robots should be able to understand and follow user instructions.

• Despite providing a powerful architecture for mapping open-vocabulary language instructions to robot actions, current vision-language-action (VLA) models struggle to follow fine-grained commands.
• One cause for this is a lack of semantic diversity and language grounding in existing robot datasets and, specifically, a lack of fine-grained task diversity for similar observations.
• To address this, we present a novel method to augment existing robot datasets by leveraging vision-language models to create counterfactual labels.