Open-World Task and Motion Planning via Vision-Language Model Generated Constraints

Understand

Foundation models like Vision-Language Models (VLMs) excel at common sense vision and language tasks such as visual question answering.

• However, they cannot yet directly solve complex, long-horizon robot manipulation problems requiring precise continuous reasoning.
• Task and Motion Planning (TAMP) systems can handle long-horizon reasoning through discrete-continuous hybrid search over parameterized skills, but rely on detailed environment models and cannot interpret novel human objectives, such as arbitrary natural language goals.
• We propose integrating VLMs into TAMP systems by having them generate discrete and continuous language-parameterized constraints that enable open-world reasoning.