Learning from Unlabeled 3D Environments for Vision-and-Language Navigation

Understand

In vision-and-language navigation (VLN), an embodied agent is required to navigate in realistic 3D environments following natural language instructions.

• One major bottleneck for existing VLN approaches is the lack of sufficient training data, resulting in unsatisfactory generalization to unseen environments.
• While VLN data is typically collected manually, such an approach is expensive and prevents scalability.
• In this work, we address the data scarcity issue by proposing to automatically create a large-scale VLN dataset from 900 unlabeled 3D buildings from HM3D.