"Where am I?" Scene Retrieval with Language

Understand

Natural language interfaces to embodied AI are becoming more ubiquitous in our daily lives.

• This opens up further opportunities for language-based interaction with embodied agents, such as a user verbally instructing an agent to execute some task in a specific location.
• For example, "put the bowls back in the cupboard next to the fridge" or "meet me at the intersection under the red sign." As such, we need methods that interface between natural language and map representations of the environment.
• To this end, we explore the question of whether we can use an open-set natural language query to identify a scene represented by a 3D scene graph.