Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts

Understand

Current Vision-and-Language Navigation (VLN) tasks mainly employ textual instructions to guide agents.

• However, being inherently abstract, the same textual instruction can be associated with different visual signals, causing severe ambiguity and limiting the transfer of prior knowledge in the vision domain from the user to the agent.
• To fill this gap, we propose Vision-and-Language Navigation with Multi-modal Prompts (VLN-MP), a novel task augmenting traditional VLN by integrating both natural language and images in instructions.
• VLN-MP not only maintains backward compatibility by effectively handling text-only prompts but also consistently shows advantages with different quantities and relevance of visual prompts.