Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity

Understand

We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon tasks over a large workspace.

• Our system is modular: it deploys state of the art Large Language Models for task planning,Vision-Language models for semantic perception, and Point Cloud transformers for grasping.
• With semantic and physical safety in mind, these modules are interfaced with a real-time trajectory optimizer and a compliant tracking controller to enable human-robot proximity.
• We demonstrate performance for the following tasks: bi-arm sorting, bottle opening, and trash disposal tasks.