RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

Understand

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully.

• In modern robotics, these capabilities are increasingly provided by vision-language models.
• However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding.
• For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives.