Just Ask:An Interactive Learning Framework for Vision and Language Navigation

Understand

In the vision and language navigation task, the agent may encounter ambiguous situations that are hard to interpret by just relying on visual information and natural language instructions.

• We propose an interactive learning framework to endow the agent with the ability to ask for users' help in such situations.
• As part of this framework, we investigate multiple learning approaches for the agent with different levels of complexity.
• The simplest model-confusion-based method lets the agent ask questions based on its confusion, relying on the predefined confidence threshold of a next action prediction model.