Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning

Understand

Grounded understanding of natural language in physical scenes can greatly benefit robots that follow human instructions.

• In object manipulation scenarios, existing end-to-end models are proficient at understanding semantic concepts, but typically cannot handle complex instructions involving spatial relations among multiple objects.
• which require both reasoning object-level spatial relations and learning precise pixel-level manipulation affordances.
• We take an initial step to this challenge with a decoupled two-stage solution.