MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning

Understand

Learning robust and scalable visual representations from massive multi-view video data remains a challenge in computer vision and autonomous driving.

• Existing pre-training methods either rely on expensive supervised learning with 3D annotations, limiting the scalability, or focus on single-frame or monocular inputs, neglecting the temporal information.
• We propose MIM4D, a novel pre-training paradigm based on dual masked image modeling (MIM).
• MIM4D leverages both spatial and temporal relations by training on masked multi-view video inputs.