3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

Understand

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network.

• In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture.
• Rather than simply projecting color data into a volumetric grid and operating solely in 3D -- which would result in insufficient detail -- we first extract feature maps from associated RGB images.
• These features are then mapped into the volumetric feature grid of a 3D network using a differentiable backprojection layer.