Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environments

Understand

Vision-and-Language Navigation in Continuous Environments (VLN-CE) is a navigation task that requires an agent to follow a language instruction in a realistic environment.

• The understanding of environments is a crucial part of the VLN-CE task, but existing methods are relatively simple and direct in understanding the environment, without delving into the relationship between language instructions and visual environments.
• Therefore, we propose a new environment representation in order to solve the above problems.
• First, we propose an Environment Representation Graph (ERG) through object detection to express the environment in semantic level.