Fetching the paper…

Reading the bibliography…

2023

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

Chen, Zhaoxi, Wang, Guangcong, Liu, Ziwei

Original
PDF
Post on X

Understand

In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise.

Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations.
At the core of SceneDreamer is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images.
Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics.

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections · Around

Built on

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and Improving the Image Quality of StyleGAN,” arXiv:1912.04958 [cs, eess, stat] , Mar. 2020, arXiv: 1912.04958. [Online]. Available: http://arxiv.org/abs/1912.04958
Original
1912
Earlier work this paper cites.
D. H. Lehmer, “Mathematical methods in large-scale computing units,” Proc. of 2nd Symp. on Large-Scale Digital Calculating Machinery , vol. 26, pp. 141–146, 1949
1949
Earlier work this paper cites.
K. Perlin, “An image synthesizer,” ACM Siggraph Computer Graphics , vol. 19, no. 3, pp. 287–296, 1985
1985
Earlier work this paper cites.
M. Olano, J. C. Hart, W. Heidrich, B. Mark, and K. Perlin, “Real-time shading languages,” in SIGGRAPH 2002 Course 36 Notes , ser. SIGGRAPH ’02, 2002
2002
Earlier work this paper cites.
E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, “pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis,” arXiv:2012.00926 [cs] , Apr. 2021, arXiv: 2012.00926. [Online]. Available: http://arxiv.org/abs/2012.00926
Original
2012
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
2014
Earlier work this paper cites.
A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, “ShapeNet: An Information-Rich 3D Model Repository,” Stanford University — Princeton University — Toyota Technological Institute at Chicago, Tech. Rep. arXiv:1512.03012 [cs.GR], 2015
Original
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
Original
2015
Earlier work this paper cites.
Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang, “Deepfashion: Powering robust clothes recognition and retrieval with rich annotations,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
Earlier work this paper cites.
M. Gadelha, S. Maji, and R. Wang, “3d shape induction from 2d views of multiple objects,” 2016. [Online]. Available: https://arxiv.org/abs/1612.05872
Original
2016
Earlier work this paper cites.
J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum, “Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling,” in Proceedings of the 30th International Conference on Neural Information Processing Systems , ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 82–90
2016
Earlier work this paper cites.
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Computer Vision – ECCV 2016 , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 694–711
2016
Earlier work this paper cites.
J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2016
2016
Earlier work this paper cites.
J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in European Conference on Computer Vision (ECCV) , 2016
2016
Earlier work this paper cites.
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proc. Computer Vision and Pattern Recognition (CVPR), IEEE , 2017
2017
Earlier work this paper cites.
J. H. Lim and J. C. Ye, “Geometric gan,” 2017. [Online]. Available: https://arxiv.org/abs/1705.02894
Original
2017
Earlier work this paper cites.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
2017
Earlier work this paper cites.
M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demystifying mmd gans,” arXiv preprint arXiv:1801.01401 , 2018
Original
2018
Earlier work this paper cites.
T. Karras, S. Laine, and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” arXiv:1812.04948 [cs, stat] , Mar. 2019, arXiv: 1812.04948. [Online]. Available: http://arxiv.org/abs/1812.04948
Original
2019
Earlier work this paper cites.
T. Nguyen-Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang, “Hologan: Unsupervised learning of 3d representations from natural images,” in The IEEE International Conference on Computer Vision (ICCV) , Nov 2019
2019
Earlier work this paper cites.
T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019
2019
Earlier work this paper cites.
J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe, “The Replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797 , 2019
Original
2019
Earlier work this paper cites.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in ECCV , 2020
2020
Earlier work this paper cites.
M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, “Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2020
2020
Earlier work this paper cites.
L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” NeurIPS , 2020
2020
Earlier work this paper cites.
K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, “Graf: Generative radiance fields for 3d-aware image synthesis,” in Advances in Neural Information Processing Systems (NeurIPS) , 2020
2020
Earlier work this paper cites.

Similar

A. Mallya, T.-C. Wang, K. Sapra, and M.-Y. Liu, “World-consistent video-to-video synthesis,” in Proceedings of the European Conference on Computer Vision , 2020
2020
Cited alongside, same era.
T. DeVries, M. A. Bautista, N. Srivastava, G. W. Taylor, and J. M. Susskind, “Unconstrained Scene Generation with Locally Conditioned Radiance Fields,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Montreal, QC, Canada: IEEE, Oct. 2021, pp. 14 284–14 293. [Online]. Available: https://ieeexplore.ieee.org/document/9710863/
2021
Cited alongside, same era.
Z. Hao, A. Mallya, S. Belongie, and M.-Y. Liu, “GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Montreal, QC, Canada: IEEE, Oct. 2021, pp. 14 052–14 062. [Online]. Available: https://ieeexplore.ieee.org/document/9710945/
2021
Cited alongside, same era.

Then

Y. Xue, Y. Li, K. K. Singh, and Y. J. Lee, “Giraffe hd: A high-resolution 3d-aware generative model,” in CVPR , 2022
2022
Later among the works it cites.
J. Gao, T. Shen, Z. Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojcic, and S. Fidler, “Get3d: A generative model of high quality 3d textured shapes learned from images,” in Advances In Neural Information Processing Systems , 2022
2022
Later among the works it cites.
Y. Deng, J. Yang, J. Xiang, and X. Tong, “Gram: Generative radiance manifolds for 3d-aware image generation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022
2022
Later among the works it cites.
J. Xiang, J. Yang, Y. Deng, and X. Tong, “GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds,” arXiv, Tech. Rep. arXiv:2206.07255, Jun. 2022, arXiv:2206.07255 [cs] type: article. [Online]. Available: http://arxiv.org/abs/2206.07255
Original
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Liu, R. Tucker, V. Jampani, A. Makadia, N. Snavely, and A. Kanazawa, “Infinite nature: Perpetual view generation of natural scenes from a single image,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021

2021

Cited alongside, same era.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelNeRF: Neural radiance fields from one or few images,” in CVPR , 2021

2021

Cited alongside, same era.

C. Reiser, S. Peng, Y. Liao, and A. Geiger, “Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps,” in International Conference on Computer Vision (ICCV) , 2021

2021

Cited alongside, same era.

A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-NeRF: Neural Radiance Fields for Dynamic Scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021

2021

Cited alongside, same era.

B. Mildenhall, P. Hedman, R. Martin-Brualla, P. Srinivasan, and J. T. Barron, “NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images,” arXiv:2111.13679 [cs, eess] , Nov. 2021, arXiv: 2111.13679. [Online]. Available: http://arxiv.org/abs/2111.13679

2021

Cited alongside, same era.

K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz, “Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields,” ACM Trans. Graph. , vol. 40, no. 6, dec 2021

2021

Cited alongside, same era.

P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” NeurIPS , 2021

2021

Cited alongside, same era.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” ICCV , 2021

2021

Cited alongside, same era.

J. Li, Z. Feng, Q. She, H. Ding, C. Wang, and G. H. Lee, “Mine: Towards continuous depth mpi with nerf for novel view synthesis,” in ICCV , 2021

2021

Cited alongside, same era.

K. Schwarz, A. Sauer, M. Niemeyer, Y. Liao, and A. Geiger, “Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids,” in Advances in Neural Information Processing Systems (NeurIPS) , 2022

2022

Later among the works it cites.

T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph. , vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022. [Online]. Available: https://doi.org/10.1145/3528223.3530127

2022

Later among the works it cites.

Z. Li, Q. Wang, N. Snavely, and A. Kanazawa, “Infinitenature-zero: Learning perpetual view generation of natural scenes from single images,” in ECCV , 2022

2022

Later among the works it cites.

M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 8248–8258

2022

Later among the works it cites.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan, “Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022

2022

Later among the works it cites.

M. Johari, Y. Lepoittevin, and F. Fleuret, “Geonerf: Generalizing nerf with geometry priors,” in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) , 2022

2022

Later among the works it cites.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” CVPR , 2022

2022

Later among the works it cites.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, “Tensorf: Tensorial radiance fields,” in European Conference on Computer Vision (ECCV) , 2022

2022

Later among the works it cites.

S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 5501–5510

2022

Later among the works it cites.

C. Sun, M. Sun, and H. Chen, “Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction,” in CVPR , 2022

2022

Later among the works it cites.

I. Skorokhodov, S. Tulyakov, Y. Wang, and P. Wonka, “Epigraf: Rethinking training of 3d gans,” arXiv preprint arXiv:2206.10535 , 2022

2022

Later among the works it cites.

K. Sun, S. Wu, Z. Huang, N. Zhang, Q. Wang, and H. Li, “Controllable 3d face synthesis with conditional generative occupancy fields,” in Advances in Neural Information Processing Systems , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022

2022

Later among the works it cites.

J. Sun, X. Wang, Y. Shi, L. Wang, J. Wang, and Y. Liu, “Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,” ACM Trans. Graph. , vol. 41, no. 6, nov 2022. [Online]. Available: https://doi.org/10.1145/3550454.3555506

2022

Later among the works it cites.

Z. Shi, Y. Shen, J. Zhu, D.-Y. Yeung, and Q. Chen, “3d-aware indoor scene synthesis with depth priors,” in ECCV , 2022

2022

Later among the works it cites.

X. Ren and X. Wang, “Look outside the room: Synthesizing a consistent long-term 3d scene video from a single image,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022

2022

Later among the works it cites.

M. A. Bautista, P. Guo, S. Abnar, W. Talbott, A. Toshev, Z. Chen, L. Dinh, S. Zhai, H. Goh, D. Ulbricht, A. Dehghan, and J. Susskind, “Gaudi: A neural architect for immersive 3d scene generation,” in NeurIPS , 2022. [Online]. Available: https://arxiv.org/abs/2207.13751

2022

Later among the works it cites.

Z. Chen, Y. Duan, W. Wang, J. He, T. Lu, J. Dai, and Y. Qiao, “Vision transformer adapter for dense predictions,” arXiv preprint arXiv:2205.08534 , 2022

2022

Later among the works it cites.

R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 3, 2022

2022

Later among the works it cites.

Z. Bylinskii, L. Herman, A. Hertzmann, S. Hutka, and Y. Zhang, “Towards better user studies in computer graphics and vision,” 2022. [Online]. Available: https://arxiv.org/abs/2206.11461

2022

Later among the works it cites.

L. Chai, R. Tucker, Z. Li, P. Isola, and N. Snavely, “Persistent nature: A generative model of unbounded 3d worlds,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023

2023

Closest in time.

C. H. Lin, H.-Y. Lee, W. Menapace, M. Chai, A. Siarohin, M.-H. Yang, and S. Tulyakov, “InfiniCity: Infinite-scale city synthesis,” arXiv preprint arXiv:2301.09637 , 2023

2023

Closest in time.