Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Understand

Large-scale training data with high-quality annotations is critical for training semantic and instance segmentation models.

• Unfortunately, pixel-wise annotation is labor-intensive and costly, raising the demand for more efficient labeling strategies.
• In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives.
• Our method utilizes NeRF as a differentiable tool to unify coarse 3D annotations and 2D semantic cues transferred from existing datasets.