Real2Code: Reconstruct Articulated Objects via Code Generation

Understand

We present Real2Code, a novel approach to reconstructing articulated objects via code generation.

• Given visual observations of an object, we first reconstruct its part geometry using an image segmentation model and a shape completion model.
• We then represent the object parts with oriented bounding boxes, which are input to a fine-tuned large language model (LLM) to predict joint articulation as code.
• By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments.