KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

Understand

Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting.

• Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner.
• Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations based on natural language instructions and visual observations of the scene.
• The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems.