ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

Understand

3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation.

• Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds.
• In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data.
• Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model.