A Recipe for Generating 3D Worlds From a Single Image

Understand

We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models.

• This approach requires minimal training and uses existing generative models.
• Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator.
• We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning.