End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

Understand

We present VLMnav, an embodied framework to transform a Vision-Language Model (VLM) into an end-to-end navigation policy.

• In contrast to prior work, we do not rely on a separation between perception, planning, and control; instead, we use a VLM to directly select actions in one step.
• Surprisingly, we find that a VLM can be used as an end-to-end policy zero-shot, i.e., without any fine-tuning or exposure to navigation data.
• This makes our approach open-ended and generalizable to any downstream navigation task.