Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Understand

Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning.

• Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning.
• However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold.
• We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly.