2020

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Schwarz, Katja, Liao, Yiyi, Niemeyer, Michael et al.

Understand

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process.

Thus, they do not provide precise control over camera viewpoint or object pose.
To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering.
However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint.

Built on

Ray tracing volume densities
J. T. Kajiya and B. V. Herzen · 1984
Earlier work this paper cites.
Optical models for direct volume rendering
N. L. Max · 1995
Earlier work this paper cites.
Cat head detection - how to effectively exploit shape and texture features
W. Zhang, J. Sun, and X. Tang · 2008
Earlier work this paper cites.
The Caltech-UCSD Birds-200-2011 Dataset
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie · 2011
Earlier work this paper cites.
Generative adversarial nets
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
D. P. Kingma and M. Welling · 2014
Earlier work this paper cites.
Learning to disentangle factors of variation with manifold interaction
S. Reed, K. Sohn, Y. Zhang, and H. Lee · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Earlier work this paper cites.
Semantic image segmentation via deep parsing network
Z. Liu, X. Li, P. Luo, C. C. Loy, and X. Tang · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
A. Radford, L. Metz, and S. Chintala · 2015
Earlier work this paper cites.
Generative and discriminative voxel modeling with convolutional neural networks
A. Brock, T. Lim, J. M. Ritchie, and N. Weston · 2016
Earlier work this paper cites.
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
X. Chen, X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel · 2016
Earlier work this paper cites.
3d-r2n2: A unified approach for single and multi-view 3d object reconstruction
C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese · 2016
Earlier work this paper cites.
Unsupervised learning of 3d structure from images
D. J. Rezende, S. M. A. Eslami, S. Mohamed, P. Battaglia, M. Jaderberg, and N. Heess · 2016
Earlier work this paper cites.
Structure-from-motion revisited
J. L. Schönberger and J.-M. Frahm · 2016
Earlier work this paper cites.
Instance normalization: The missing ingredient for fast stylization
D. Ulyanov, A. Vedaldi, and V. S. Lempitsky · 2016
Earlier work this paper cites.
Generative image modeling using style and structure adversarial networks
X. Wang and A. Gupta · 2016
Earlier work this paper cites.
Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling
J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum · 2016
Earlier work this paper cites.
CARLA: An open urban driving simulator
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter · 2017
Earlier work this paper cites.
Image-to-image translation with conditional adversarial networks
P. Isola, J. Zhu, T. Zhou, and A. A. Efros · 2017
Earlier work this paper cites.
Octnet: Learning deep 3d representations at high resolutions
G. Riegler, A. O. Ulusoy, and A. Geiger · 2017
Earlier work this paper cites.
Interpretable transformations with encoder-decoder networks
D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow · 2017
Earlier work this paper cites.
Learning representations and generative models for 3d point clouds
P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. J. Guibas · 2018
Earlier work this paper cites.
Geometric image synthesis
H. Alhaija, S. Mustikovela, A. Geiger, and C. Rother · 2018
Earlier work this paper cites.
Demystifying MMD gans
M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton · 2018
Earlier work this paper cites.

Similar

AtlasNet: A papier-mâché approach to learning 3d surface generation
T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry · 2018
Cited alongside, same era.
Progressive growing of GANs for improved quality, stability, and variation
T. Karras, T. Aila, S. Laine, and J. Lehtinen · 2018
Cited alongside, same era.
Deep marching cubes: Learning explicit surface representations
Y. Liao, S. Donne, and A. Geiger · 2018
Cited alongside, same era.
Detection of gan-generated fake images over social networks
F. Marra, D. Gragnaniello, D. Cozzolino, and L. Verdoliva · 2018
Cited alongside, same era.
Which training methods for gans do actually converge?
L. Mescheder, A. Geiger, and S. Nowozin · 2018
Cited alongside, same era.
Spectral normalization for generative adversarial networks
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida · 2018
Cited alongside, same era.
Photoshape: Photorealistic materials for large-scale shape collections
K. Park, K. Rematas, A. Farhadi, and S. M. Seitz · 2018
Cited alongside, same era.
Unsupervised geometry-aware representation for 3d human pose estimation
H. Rhodin, M. Salzmann, and P. Fua · 2018
Cited alongside, same era.
Pixel2Mesh: Generating 3D mesh models from single RGB images
N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang · 2018
Cited alongside, same era.
Modular generative adversarial networks
B. Zhao, B. Chang, Z. Jie, and L. Sigal · 2018
Cited alongside, same era.
Stereo magnification: learning view synthesis using multiplane images
T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely · 2018
Cited alongside, same era.
Visual object networks: Image generation with disentangled 3d representations
J. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and B. Freeman · 2018
Cited alongside, same era.

Then

Texture fields: Learning texture representations in function space
M. Oechsle, L. Mescheder, M. Niemeyer, T. Strauss, and A. Geiger · 2019
Later among the works it cites.
Deep mesh reconstruction from single rgb images via topology modification networks
J. Pan, X. Han, W. Chen, J. Tang, and K. Jia · 2019
Later among the works it cites.
Deepsdf: Learning continuous signed distance functions for shape representation
J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove · 2019
Later among the works it cites.
On the spectral bias of neural networks
N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. C. Courville · 2019
Later among the works it cites.
Generating diverse high-fidelity images with VQ-VAE-2
A. Razavi, A. van den Oord, and O. Vinyals · 2019
Later among the works it cites.
Faceforensics++: Learning to detect manipulated facial images
A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner · 2019
Later among the works it cites.
Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization
S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li · 2019
Later among the works it cites.
Deepvoxels: Learning persistent 3d feature embeddings
V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer · 2019
Later among the works it cites.
Scene representation networks: Continuous 3d-structure-aware neural scene representations
V. Sitzmann, M. Zollhöfer, and G. Wetzstein · 2019
Later among the works it cites.
Monocular neural image based rendering with continuous view control
J. Song, X. Chen, and O. Hilliges · 2019
Later among the works it cites.
Pushing the boundaries of view extrapolation with multiplane images
P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely · 2019
Later among the works it cites.
Pix2vox: Context-aware 3d reconstruction from single and multi-view images
H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang · 2019
Later among the works it cites.
Deep local shapes: Learning local SDF priors for detailed 3d reconstruction
R. Chabra, J. E. Lenssen, E. Ilg, T. Schmidt, J. Straub, S. Lovegrove, and R. A. Newcombe · 2020
Closest in time.
Local implicit grid representations for 3d scenes
C. Jiang, A. Sud, A. Makadia, J. Huang, M. NieBner, and T. Funkhouser · 2020
Closest in time.
Analyzing and improving the image quality of StyleGAN
T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila · 2020
Closest in time.
High-fidelity synthesis with disentangled representation
W. Lee, D. Kim, S. Hong, and H. Lee · 2020
Closest in time.
Towards unsupervised learning of generative models for 3d controllable image synthesis
Y. Liao, K. Schwarz, L. Mescheder, and A. Geiger · 2020
Closest in time.
Nerf: Representing scenes as neural radiance fields for view synthesis
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng · 2020
Closest in time.
Blockgan: Learning 3d object-aware scene representations from unlabelled images
T. Nguyen-Phuoc, C. Richardt, L. Mai, Y. Yang, and N. J. Mitra · 2020
Closest in time.
Semi-supervised stylegan for disentanglement learning
W. Nie, T. Karras, A. Garg, S. Debhath, A. Patney, A. B. Patel, and A. Anandkumar · 2020
Closest in time.
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger · 2020
Closest in time.
RGBD-GAN: unsupervised 3d representation learning from natural image datasets via RGBD image synthesis
A. Noguchi and T. Harada · 2020
Closest in time.
Learning implicit surface light fields
M. Oechsle, M. Niemeyer, L. Mescheder, T. Strauss, and A. Geiger · 2020
Closest in time.
Convolutional occupancy networks
S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger · 2020
Closest in time.
Universal differentiable renderer for implicit neural representations
L. Yariv, M. Atzmon, and Y. Lipman · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…