CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP

Understand

Training a 3D scene understanding model requires complicated human annotations, which are laborious to collect and result in a model only encoding close-set object semantics.

• In contrast, vision-language pre-training models (e.g., CLIP) have shown remarkable open-world reasoning properties.
• To this end, we propose directly transferring CLIP's feature space to 3D scene understanding model without any form of supervision.
• We first modify CLIP's input and forwarding process so that it can be adapted to extract dense pixel features for 3D scene contents.