Panoptic Lifting for 3D Scene Understanding with Neural Fields

Understand

We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes.

• Once trained, our model can render color images together with 3D-consistent panoptic segmentation from novel viewpoints.
• Unlike existing approaches which use 3D input directly or indirectly, our method requires only machine-generated 2D panoptic segmentation masks inferred from a pre-trained network.
• Our core contribution is a panoptic lifting scheme based on a neural field representation that generates a unified and multi-view consistent, 3D panoptic representation of the scene.