2023

ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation

Wang, Peng, Shi, Yichun

Understand

We introduce "ImageDream," an innovative image-prompt, multi-view diffusion model for 3D object generation.

ImageDream stands out for its ability to produce 3D models of higher quality compared to existing state-of-the-art, image-conditioned methods.
Our approach utilizes a canonical camera coordination for the objects in images, improving visual geometry accuracy.
The model is designed with various levels of control at each block inside the diffusion model based on the input image, where global control shapes the overall object layout and local control fine-tunes the image details.

Built on

Auto-encoding variational bayes
Diederik P Kingma and Max Welling · 2014
Earlier work this paper cites.
Shapenet: An information-rich 3d model repository
Original
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al · 2015
Earlier work this paper cites.
Improved techniques for training gans
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen · 2016
Earlier work this paper cites.
Hologan: Unsupervised learning of 3d representations from natural images
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang · 2019
Earlier work this paper cites.
Scene representation networks: Continuous 3d-structure-aware neural scene representations
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein · 2019
Earlier work this paper cites.
Learning single-image 3d reconstruction by generative modelling of shape, pose and shading
Paul Henderson and Vittorio Ferrari · 2020
Earlier work this paper cites.
Leveraging 2d data to learn textured 3d mesh generation
Paul Henderson, Vagia Tsiminaki, and Christoph H Lampert · 2020
Earlier work this paper cites.
Blockgan: Learning 3d object-aware scene representations from unlabelled images
Thu H Nguyen-Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra · 2020
Earlier work this paper cites.
Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2021
Earlier work this paper cites.
Giraffe: Representing scenes as compositional generative neural feature fields
Michael Niemeyer and Andreas Geiger · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Earlier work this paper cites.
Efficient geometry-aware 3d generative adversarial networks
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al · 2022
Earlier work this paper cites.
Gram: Generative radiance manifolds for 3d-aware image generation
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong · 2022
Earlier work this paper cites.

Similar

Get3d: A generative model of high quality 3d textured shapes learned from images
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler · 2022
Cited alongside, same era.
Point-e: A system for generating 3d point clouds from complex prompts
Original
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen · 2022
Cited alongside, same era.
Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs
Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan · 2022
Cited alongside, same era.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Cited alongside, same era.
Laion-5b: An open large-scale dataset for training next generation image-text models
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al · 2022
Cited alongside, same era.
https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0
stable-diffusion-xl-base-1.0 · 2023
Cited alongside, same era.
GeNVS: Generative novel view synthesis with 3D-aware diffusion models
Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein · 2023
Cited alongside, same era.
Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation
Original
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia · 2023
Cited alongside, same era.
Dreamtime: An improved optimization strategy for text-to-3d content creation
Original
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang · 2023
Cited alongside, same era.
Shap-e: Generating conditional 3d implicit functions
Original
Heewoo Jun and Alex Nichol · 2023
Cited alongside, same era.
Holodiffusion: Training a 3d diffusion model using 2d images
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra · 2023
Cited alongside, same era.
Wonder3d: Single image to 3d using cross-domain diffusion, 2023
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang · 2023
Cited alongside, same era.

Then

Realfusion: 360deg reconstruction of any object from a single image
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi · 2023
Closest in time.
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Original
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach · 2023
Closest in time.
Dreamfusion: Text-to-3d using 2d diffusion
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall · 2023
Closest in time.
Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors
Original
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem · 2023
Closest in time.
3d neural field generation using triplane diffusion
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein · 2023
Closest in time.
Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior, 2023
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu · 2023
Closest in time.
Viewset diffusion:(0-) image-conditioned 3d generative models from 2d data, 2023
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi · 2023
Closest in time.
Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior
Original
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen · 2023
Closest in time.
Textmesh: Generation of realistic 3d meshes from text prompts
Original
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari · 2023
Closest in time.
Novel view synthesis with diffusion models
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi · 2023
Closest in time.
Multiview compressive coding for 3d reconstruction
Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari · 2023
Closest in time.
Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction
Zhizhuo Zhou and Shubham Tulsiani · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…