SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

Understand

Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks.

• However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics.
• We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations.
• To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures.