World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

Understand

The Multi-modal Large Language Models (MLLMs) with extensive world knowledge have revitalized autonomous driving, particularly in reasoning tasks within perceivable regions.

• However, when faced with perception-limited areas (dynamic or static occlusion regions), MLLMs struggle to effectively integrate perception ability with world knowledge for reasoning.
• These perception-limited regions can conceal crucial safety information, especially for vulnerable road users.
• In this paper, we propose a framework, which aims to improve autonomous driving performance under perceptionlimited conditions by enhancing the integration of perception capabilities and world knowledge.