2021

PlenOctrees for Real-time Rendering of Neural Radiance Fields

Yu, Alex, Li, Ruilong, Tancik, Matthew et al.

Understand

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects.

Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs.
We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects.
Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree.

Built on

Dispersion on a sphere
Ronald Aylmer Fisher · 1953
Earlier work this paper cites.
Color image quantization for frame buffer display
Paul Heckbert · 1982
Earlier work this paper cites.
Volume rendering
Robert A Drebin, Loren Carpenter, and Pat Hanrahan · 1988
Earlier work this paper cites.
The plenoptic function and the elements of early vision
Edward H Adelson, James R Bergen, et al · 1991
Earlier work this paper cites.
Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach
Paul E Debevec, Camillo J Taylor, and Jitendra Malik · 1996
Earlier work this paper cites.
Light field rendering
Marc Levoy and Pat Hanrahan · 1996
Earlier work this paper cites.
Layered depth images
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski · 1998
Earlier work this paper cites.
Stereo matching with transparency and matting
Richard Szeliski and Polina Golland · 1998
Earlier work this paper cites.
Photorealistic scene reconstruction by voxel coloring
Steven M Seitz and Charles R Dyer · 1999
Earlier work this paper cites.
A theory of shape by space carving
Kiriakos N Kutulakos and Steven M Seitz · 2000
Earlier work this paper cites.
Surface light fields for 3d photography
Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle · 2000
Earlier work this paper cites.
Unstructured lumigraph rendering
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen · 2001
Earlier work this paper cites.
On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object
Ravi Ramamoorthi and Pat Hanrahan · 2001
Earlier work this paper cites.
Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments
Peter-Pike Sloan, Jan Kautz, and John Snyder · 2002
Earlier work this paper cites.
Lambertian reflectance and linear subspaces
Ronen Basri and David W Jacobs · 2003
Earlier work this paper cites.
A survey of octree volume rendering methods
Aaron Knoll · 2006
Earlier work this paper cites.
All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation
Yu-Ting Tsai and Zen-Chung Shih · 2006
Earlier work this paper cites.
Let there be color! large-scale texturing of 3d reconstructions
Michael Waechter, Nils Moehrle, and Michael Goesele · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Hierarchical surface prediction for 3d object reconstruction
Christian Häne, Shubham Tulsiani, and Jitendra Malik · 2017
Earlier work this paper cites.
Learning a multi-view stereo machine
Abhishek Kar, Christian Häne, and Jitendra Malik · 2017
Earlier work this paper cites.
Tanks and temples: Benchmarking large-scale scene reconstruction
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun · 2017
Earlier work this paper cites.

Similar

zlib, 2017
Jean loup Gailly and Mark Adler · 2017
Cited alongside, same era.
Soft 3d reconstruction for view synthesis
Eric Penner and Li Zhang · 2017
Cited alongside, same era.
Octnet: Learning deep 3d representations at high resolutions
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger · 2017
Cited alongside, same era.
Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox · 2017
Cited alongside, same era.
Multi-view supervision for single-view reconstruction via differentiable ray consistency
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik · 2017
Cited alongside, same era.
O-cnn: Octree-based convolutional neural networks for 3d shape analysis
Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong · 2017
Cited alongside, same era.
Stereo magnification: Learning view synthesis using multiplane images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely · 2018
Cited alongside, same era.
Learning implicit fields for generative shape modeling
Zhiqin Chen and Hao Zhang · 2019
Cited alongside, same era.
Neural volumes: Learning dynamic renderable volumes from images
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh · 2019
Cited alongside, same era.
Occupancy networks: Learning 3d reconstruction in function space
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger · 2019
Cited alongside, same era.
Local light field fusion: Practical view synthesis with prescriptive sampling guidelines
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar · 2019
Cited alongside, same era.
Deepsdf: Learning continuous signed distance functions for shape representation
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove · 2019
Cited alongside, same era.

Then

Neural sparse voxel fields
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt · 2020
Later among the works it cites.
NeRF: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Later among the works it cites.
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger · 2020
Later among the works it cites.
Convolutional occupancy networks
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger · 2020
Later among the works it cites.
GRAF: Generative radiance fields for 3d-aware image synthesis
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger · 2020
Later among the works it cites.
Multiview neural surface reconstruction by disentangling geometry and appearance
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman · 2020
Later among the works it cites.
Fastnerf: High-fidelity neural rendering at 200fps
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin · 2021
Closest in time.
Baking neural radiance fields for real-time view synthesis
Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec · 2021
Closest in time.
Autoint: Automatic integration for fast neural volume rendering
David B Lindell, Julien NP Martel, and Gordon Wetzstein · 2021
Closest in time.
Mixture of volumetric primitives for efficient neural rendering
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih · 2021
Closest in time.
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth · 2021
Closest in time.
Deformable neural radiance fields
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo-Martin Brualla · 2021
Closest in time.
DeRF: Decomposed radiance fields
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi · 2021
Closest in time.
Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger · 2021
Closest in time.
NeRV: Neural reflectance and visibility fields for relighting and view synthesis
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron · 2021
Closest in time.
Neural geometric level of detail: Real-time rendering with implicit 3D shapes
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler · 2021
Closest in time.
Learned initializations for optimizing coordinate-based neural representations
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng · 2021
Closest in time.
GRF: Learning a general radiance field for 3d scene representation and rendering
Alex Trevithick and Bo Yang · 2021
Closest in time.
IBRNet: Learning multi-view image-based rendering
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser · 2021
Closest in time.
NeX: Real-time view synthesis with neural basis expansion
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn · 2021
Closest in time.
pixelNeRF: Neural radiance fields from one or few images
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…