One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones

Understand

We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task.

• Significant progress has been made in recent years, especially for tasks with short horizons.
• However, when it comes to long-horizon tasks with extended sequences of actions, an agent can easily ignore some instructions or get stuck in the middle of the long instructions and eventually fail the task.
• To address this challenge, we propose a model-agnostic milestone-based task tracker (M-TRACK) to guide the agent and monitor its progress.