RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

Understand

3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR).

• Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries.
• Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem.
• Specifically, given a single RGB-D image and a novel viewpoint (encoded as a collection of query rays), we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays.