Multi-Agent Planning Using Visual Language Models

Understand

Large Language Models (LLMs) and Visual Language Models (VLMs) are attracting increasing interest due to their improving performance and applications across various domains and tasks.

• However, LLMs and VLMs can produce erroneous results, especially when a deep understanding of the problem domain is required.
• For instance, when planning and perception are needed simultaneously, these models often struggle because of difficulties in merging multi-modal information.
• To address this issue, fine-tuned models are typically employed and trained on specialized data structures representing the environment.