Multimodal Grounding for Embodied AI via Augmented Reality Headsets for Natural Language Driven Task Planning

Understand

Recent advances in generative modeling have spurred a resurgence in the field of Embodied Artificial Intelligence (EAI).

• EAI systems typically deploy large language models to physical systems capable of interacting with their environment.
• In our exploration of EAI for industrial domains, we successfully demonstrate the feasibility of co-located, human-robot teaming.
• Specifically, we construct an experiment where an Augmented Reality (AR) headset mediates information exchange between an EAI agent and human operator for a variety of inspection tasks.