Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models

Understand

Foundation models have achieved remarkable results in 2D and language tasks like image segmentation, object detection, and visual-language understanding.

• However, their potential to enrich 3D scene representation learning is largely untapped due to the existence of the domain gap.
• In this work, we propose an innovative methodology called Bridge3D to address this gap by pre-training 3D models using features, semantic masks, and captions sourced from foundation models.
• Specifically, our method employs semantic masks from foundation models to guide the masking and reconstruction process for the masked autoencoder, enabling more focused attention on foreground representations.