USA-Net: Unified Semantic and Affordance Representations for Robot Memory

Understand

In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of their environment.

• Robotic systems often handle these through separate pipelines, sometimes using very different representation spaces, which can be suboptimal when the two objectives conflict.
• In this work, we present USA-Net, a simple method for constructing a world representation that encodes both the semantics and spatial affordances of a scene in a differentiable map.
• This allows us to build a gradient-based planner which can navigate to locations in the scene specified using open-ended vocabulary.