2022

DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis

Xu, Yinghao, Chai, Menglei, Shi, Zifan et al.

Understand

Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects.

This work presents DisCoScene: a 3Daware generative model for high-quality and controllable scene synthesis.
The key ingredient of our method is a very abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background.
Moreover, it serves as an intuitive user control for scene editing.

Built on

Scene graphs in the new millennium
Henry Sowizral · 2000
Earlier work this paper cites.
Lessons from scene graphs: using scene graphs to teach hierarchical modeling
Steve Cunningham and Michael J Bailey · 2001
Earlier work this paper cites.
Image parsing: Unifying segmentation, detection, and recognition
Zhuowen Tu, Xiangrong Chen, Alan L Yuille, and Song-Chun Zhu · 2005
Earlier work this paper cites.
Game Graphic Programming
A. Sherrod · 2008
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Earlier work this paper cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick · 2017
Earlier work this paper cites.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros · 2017
Earlier work this paper cites.
Demystifying mmd gans
Original
Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton · 2018
Earlier work this paper cites.
Image generation from scene graphs
Justin Johnson, Agrim Gupta, and Li Fei-Fei · 2018
Earlier work this paper cites.
Progressive growing of gans for improved quality, stability, and variation
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2018
Earlier work this paper cites.
High-resolution image synthesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro · 2018
Earlier work this paper cites.
Visual object networks: Image generation with disentangled 3D representations
Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, and William T. Freeman · 2018
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Layoutgan: Generating graphic layouts with wireframe discriminators
Original
Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu · 2019
Earlier work this paper cites.
Occupancy networks: Learning 3d reconstruction in function space
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger · 2019
Earlier work this paper cites.
Hologan: Unsupervised learning of 3d representations from natural images
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang · 2019
Earlier work this paper cites.
Deepsdf: Learning continuous signed distance functions for shape representation
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove · 2019
Earlier work this paper cites.
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Earlier work this paper cites.
Learning physical graph representations from visual scenes
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li F Fei-Fei, Jiajun Wu, Josh Tenenbaum, et al · 2020
Earlier work this paper cites.
Object-centric neural scene rendering
Original
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser · 2020
Earlier work this paper cites.
Analyzing and improving the image quality of StyleGAN
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila · 2020
Earlier work this paper cites.

Similar

Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Cited alongside, same era.
Blockgan: Learning 3d object-aware scene representations from unlabelled images
Thu H Nguyen-Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra · 2020
Cited alongside, same era.
Graf: Generative radiance fields for 3d-aware image synthesis
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger · 2020
Cited alongside, same era.
Interfacegan: Interpreting the disentangled face representation learned by gans
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou · 2020
Cited alongside, same era.
Scalability in perception for autonomous driving: Waymo open dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al · 2020
Cited alongside, same era.
Nerf++: Analyzing and improving neural radiance fields
Original
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun · 2020
Cited alongside, same era.
Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan · 2021
Cited alongside, same era.
pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein · 2021
Cited alongside, same era.
Unconstrained scene generation with locally conditioned radiance fields
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind · 2021
Cited alongside, same era.
3d-front: 3d furnished rooms with layouts and semantics
Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al · 2021
Cited alongside, same era.
3d-future: 3d furniture shape with texture
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao · 2021
Cited alongside, same era.
Detail me more: Improving gan’s photo-realism of complex scenes
Raghudeep Gadde, Qianli Feng, and Aleix M Martinez · 2021
Cited alongside, same era.

Then

Generative occupancy fields for 3d surface-aware image synthesis
Xudong Xu, Xingang Pan, Dahua Lin, and Bo Dai · 2021
Later among the works it cites.
Generative hierarchical features from synthesizing images
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou · 2021
Later among the works it cites.
Learning object-compositional neural radiance field for editable scene rendering
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui · 2021
Later among the works it cites.
Semantic hierarchy emerges in deep generative representations for scene synthesis
Ceyuan Yang, Yujun Shen, and Bolei Zhou · 2021
Later among the works it cites.
Decorating your own bedroom: Locally controlling image generation with generative adversarial networks
Chen Zhang, Yinghao Xu, and Yujun Shen · 2021
Later among the works it cites.
Efficient geometry-aware 3d generative adversarial networks
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al · 2022
Closest in time.
Gram: Generative radiance manifolds for 3d-aware image generation
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong · 2022
Closest in time.
Blobgan: Spatially disentangled scene representations
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A. Efros · 2022
Closest in time.
Get3d: A generative model of high quality 3d textured shapes learned from images
Original
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler · 2022
Closest in time.
Stylesdf: High-resolution 3d-consistent image and geometry generation
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman · 2022
Closest in time.
Pivotal tuning for latent-based editing of real images
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or · 2022
Closest in time.
Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger · 2022
Closest in time.
3d-aware indoor scene synthesis with depth priors
Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, and Qifeng Chen · 2022
Closest in time.
Improving 3d-aware image synthesis with a geometry-aware discriminator
Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, and Dit-Yan Yeung · 2022
Closest in time.
Epigraf: Rethinking training of 3d gans
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka · 2022
Closest in time.
Semantic probability distribution modeling for diverse semantic image synthesis
Zhentao Tan, Qi Chu, Menglei Chai, Dongdong Chen, Jing Liao, Qiankun Liu, Bin Liu, Gang Hua, and Nenghai Yu · 2022
Closest in time.
Block-nerf: Scalable large scene neural view synthesis
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar · 2022
Closest in time.
Improving gan equilibrium by raising spatial awareness
Jianyuan Wang, Ceyuan Yang, Yinghao Xu, Yujun Shen, Hongdong Li, and Bolei Zhou · 2022
Closest in time.
Object-compositional neural implicit surfaces
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng · 2022
Closest in time.
Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering
Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin · 2022
Closest in time.
3d-aware image synthesis via learning structural and textural representations
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou · 2022
Closest in time.
Giraffe hd: A high-resolution 3d-aware generative model
Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jae Lee · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…