DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Understand

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving.

• While prevalent multi-modal methods simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance.
• However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities.
• In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion.