SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

Understand

Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications.

• While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space, i.e., reasoning about the effect of egocentric and object motions on spatial relationships.
• Manually annotating such object and camera movements is expensive.
• Hence, we introduce SAT, a simulated spatial aptitude training dataset utilizing 3D simulators, comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes.