PaLM-E: An Embodied Multimodal Language Model

Understand

Large language models excel at a wide range of complex tasks.

• However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding.
• We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts.
• Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings.