Language-guided Adaptive Perception with Hierarchical Symbolic Representations for Mobile Manipulators

Understand

Language is an effective medium for bi-directional communication in human-robot teams.

• To infer the meaning of many instructions, robots need to construct a model of their surroundings that describe the spatial, semantic, and metric properties of objects from observations and prior information about the environment.
• Recent algorithms condition the expression of object detectors in a robot's perception pipeline on language to generate a minimal representation of the environment necessary to efficiently determine the meaning of the instruction.
• We expand on this work by introducing the ability to express hierarchies between detectors.