2024

ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance

Chen, Yongwei, Wang, Tengfei, Wu, Tong et al.

Understand

Generating high-quality 3D assets from a given image is highly desirable in various applications such as AR/VR.

Recent advances in single-image 3D generation explore feed-forward models that learn to infer the 3D model of an object without optimization.
Though promising results have been achieved in single object generation, these methods often struggle to model complex 3D assets that inherently contain multiple objects.
In this work, we present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models.

Built on

Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal, A., Rusinkiewicz, S., Dobkin, D.: Modeling by example. ACM transactions on graphics (TOG) 23
2004
Earlier work this paper cites.
Kalogerakis, E., Chaudhuri, S., Koller, D., Koltun, V.: A probabilistic model for component-based shape synthesis. Acm Transactions on Graphics (TOG) 31
2012
Earlier work this paper cites.
Chaudhuri, S., Kalogerakis, E., Giguere, S., Funkhouser, T.: Attribit: content creation with semantic attributes. In: Proceedings of the 26th annual ACM symposium on User interface software and technology. pp. 193–202 (2013)
2013
Earlier work this paper cites.
Schulz, A., Shamir, A., Levin, D.I.W., Sitthi-Amorn, P., Matusik, W.: Design and fabrication by example. ACM Transactions on Graphics (Proceedings SIGGRAPH 2014) 33
2014
Earlier work this paper cites.
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33
2020
Earlier work this paper cites.
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
2020
Earlier work this paper cites.
Ravi, N., Reizenstein, J., Novotny, D., Gordon, T., Lo, W.Y., Johnson, J., Gkioxari, G.: Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501 (2020)
Original
2020
Earlier work this paper cites.
Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5799–5809 (2021)
2021
Earlier work this paper cites.
Niemeyer, M., Geiger, A.: Giraffe: Representing scenes as compositional generative neural feature fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11453–11464 (2021)
2021
Earlier work this paper cites.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
2021
Earlier work this paper cites.
Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. ICCV (2021)
2021
Earlier work this paper cites.
Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems 34
2021
Earlier work this paper cites.
Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., et al.: Efficient geometry-aware 3d generative adversarial networks. In: CVPR (2022)
2022
Earlier work this paper cites.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. https://arxiv.org/abs/2212.08051 (2022)
Original
2022
Earlier work this paper cites.
Gu, J., Liu, L., Wang, P., Theobalt, C.: Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. In: ICLR (2022)
2022
Earlier work this paper cites.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
Original
2022
Earlier work this paper cites.
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., Cohen-Or, D.: Latent-nerf for shape-guided generation of 3d shapes and textures. https://arxiv.org/abs/2211.07600 (2022)
Original
2022
Earlier work this paper cites.
Or-El, R., Luo, X., Shan, M., Shechtman, E., Park, J.J., Kemelmacher-Shlizerman, I.: Stylesdf: High-resolution 3d-consistent image and geometry generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13503–13513 (2022)
2022
Earlier work this paper cites.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
Earlier work this paper cites.
Willis, K.D., Jayaraman, P.K., Chu, H., Tian, Y., Li, Y., Grandi, D., Sanghi, A., Tran, L., Lambourne, J.G., Solar-Lezama, A., et al.: Joinable: Learning bottom-up assembly of parametric cad joints. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15849–15860 (2022)
2022
Earlier work this paper cites.
Xiang, J., Yang, J., Deng, Y., Tong, X.: Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. arXiv preprint arXiv:2206.07255 (2022)
Original
2022
Earlier work this paper cites.

Similar

Chen, R., Chen, Y., Jiao, N., Jia, K.: Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3d content creation. https://arxiv.org/abs/2303.13873 (2023)
Original
2023
Cited alongside, same era.
Cheng, Y.C., Lee, H.Y., Tulyakov, S., Schwing, A., Gui, L.: SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
Cited alongside, same era.
Chou, G., Bahat, Y., Heide, F.: Diffusion-SDF: Conditional generative modeling of signed distance functions. https://arxiv.org/abs/2211.13757 (2023)
Original
2023
Cited alongside, same era.
Gao, G., Liu, W., Chen, A., Geiger, A., Schölkopf, B.: Graphdreamer: Compositional 3d scene synthesis from scene graphs. arXiv preprint arXiv:2312.00093 (2023)
Original
2023
Cited alongside, same era.
Gupta, A., Xiong, W., Nie, Y., Jones, I., Oğuz, B.: 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371 (2023)
Original
2023
Cited alongside, same era.
He, Z., Wang, T.: Openlrm: Open-source large reconstruction models. https://github.com/3DTopia/OpenLRM (2023)
2023
Cited alongside, same era.
Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023)
Original
2023
Cited alongside, same era.
Jun, H., Nichol, A.: Shap-e: Generating conditional 3d implicit functions. https://arxiv.org/abs/2305.02463 (2023)
Original
2023
Cited alongside, same era.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv:2304.02643 (2023)
Original
2023
Cited alongside, same era.
Li, J., Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong, Y., Sunkavalli, K., Shakhnarovich, G., Bi, S.: Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model (2023)
2023
Cited alongside, same era.
Li, W., Chen, R., Chen, X., Tan, P.: SweetDreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. https://arxiv.org/abs/2310.02596 (2023)
Original
2023
Cited alongside, same era.
Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3D: High-resolution text-to-3d content creation. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
Cited alongside, same era.

Then

Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3d using 2d diffusion. In: International Conference on Learning Representations (ICLR) (2023)
2023
Later among the works it cites.
Qian, G., Mai, J., Hamdi, A., Ren, J., Siarohin, A., Li, B., Lee, H.Y., Skorokhodov, I., Wonka, P., Tulyakov, S., Ghanem, B.: Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. https://arxiv.org/abs/2306.17843 (2023)
Original
2023
Later among the works it cites.
Sun, J., Zhang, B., Shao, R., Wang, L., Liu, W., Xie, Z., Liu, Y.: Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. https://arxiv.org/abs/2310.16818 (2023)
Original
2023
Later among the works it cites.
Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation (2023)
2023
Later among the works it cites.
Tang, J., Wang, T., Zhang, B., Zhang, T., Yi, R., Ma, L., Chen, D.: Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In: International Conference on Computer Vision ICCV (2023)
2023
Later among the works it cites.
Tertikas, K., Despoina, P., Pan, B., Park, J.J., Uy, M.A., Emiris, I., Avrithis, Y., Guibas, L.: Partnerf: Generating part-aware editable 3d shapes without 3d supervision. arXiv preprint arXiv:2303.09554 (2023)
Original
2023
Later among the works it cites.
Vilesov, A., Chari, P., Kadambi, A.: Cg3d: Compositional generation for text-to-3d via gaussian splatting. arXiv preprint arXiv:2311.17907 (2023)
Original
2023
Later among the works it cites.
Wang, P., Shi, Y.: Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 (2023)
Original
2023
Later among the works it cites.
Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., Guo, B.: Rodin: A generative model for sculpting 3d digital avatars using diffusion. https://arxiv.org/abs/2212.06135 (2023)
Original
2023
Later among the works it cites.
Wang, Z., Li, M., Chen, C.: Luciddreaming: Controllable object-centric 3d generation. arXiv preprint arXiv:2312.00588 (2023)
Original
2023
Later among the works it cites.
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: ProlificDreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. https://arxiv.org/abs/2305.16213 (2023)
Original
2023
Later among the works it cites.
Wu, T., Yang, G., Li, Z., Zhang, K., Liu, Z., Guibas, L., Lin, D., Wetzstein, G.: Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092 (2023)
Original
2023
Later among the works it cites.
Xu, D., Jiang, Y., Wang, P., Fan, Z., Wang, Y., Wang, Z.: NeuralLift-360: Lifting an in-the-wild 2d photo to a 3d object with 360 views. https://arxiv.org/abs/2211.16431 (2023)
Original
2023
Later among the works it cites.
Yi, T., Fang, J., Wu, G., Xie, L., Zhang, X., Liu, W., Tian, Q., Wang, X.: GaussianDreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. https://arxiv.org/abs/2310.08529 (2023)
Original
2023
Later among the works it cites.
Yichun, S., Peng, W., Jianglong, Y., Long, M., Kejie, L., Xiao, Y.: MVDream: Multi-view diffusion for 3d generation. https://arxiv.org/abs/2308.16512 (2023)
Original
2023
Later among the works it cites.
Zhao, Z., Liu, W., Chen, X., Zeng, X., Wang, R., Cheng, P., FU, B., Chen, T., YU, G., Gao, S.: Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
2023
Later among the works it cites.
Epstein, D., Poole, B., Mildenhall, B., Efros, A.A., Holynski, A.: Disentangled 3d scene generation with layout learning. arXiv preprint arXiv:2402.16936 (2024)
Original
2024
Closest in time.
Hong, F., Tang, J., Cao, Z., Shi, M., Wu, T., Chen, Z., Wang, T., Pan, L., Lin, D., Liu, Z.: 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint arXiv:2403.02234 (2024)
Original
2024
Closest in time.
Tang, J., Chen, Z., Chen, X., Wang, T., Zeng, G., Liu, Z.: Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 (2024)
Original
2024
Closest in time.
Tang, S., Chen, J., Wang, D., Tang, C., Zhang, F., Fan, Y., Chandra, V., Furukawa, Y., Ranjan, R.: Mvdiffusion++: A dense high-resolution multi-view diffusion model for single to sparse-view 3d objectreconstruction. arXiv preprint arXiv:2402.12712 (2024)
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…