VLAs are Confined yet Capable of Generalizing to Novel Instructions

Understand

Vision-language-action models (VLAs) often achieve high performance on demonstrated tasks but struggle significantly when required to extrapolate, combining skills learned from different tasks in novel ways.

• For instance, VLAs might successfully put the cream cheese in the bowl and put the bowl on top of the cabinet, yet still fail to put the cream cheese on top of the cabinet.
• In this work, we demonstrate that behaviors from distinct tasks can be effectively recombined by manipulating the VLA's internal representations at inference time.
• Concretely, we identify the text latent by averaging the text tokens' hidden states across all demonstrated trajectories for a specific base task.