Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

Understand

Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing.

• This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks.
• Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks.
• In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them.