3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

Understand

Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining.

• However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability.
• In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance.
• Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models.