2023

pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction

Charatan, David, Li, Sizhe, Tagliasacchi, Andrea et al.

Understand

We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D radiance fields parameterized by 3D Gaussian primitives from pairs of images.

Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time.
To overcome local minima inherent to sparse and locally supported representations, we predict a dense probability distribution over 3D and sample Gaussian means from that probability distribution.
We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation.

Built on

Maximum likelihood from incomplete data via the em algorithm
Arthur P Dempster, Nan M Laird, and Donald B Rubin · 1977
Earlier work this paper cites.
Image quality assessment: From error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli · 2004
Earlier work this paper cites.
Depth map prediction from a single image using a multi-scale deep network
David Eigen, Christian Puhrsch, and Rob Fergus · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Diederik P Kingma and Max Welling · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin J Wainwright, and Michael I Jordan · 2016
Earlier work this paper cites.
Rendernet: A deep convolutional network for differentiable rendering from 3d shapes
Thu H Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang · 2018
Earlier work this paper cites.
The unreasonable effectiveness of deep networks as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
Stereo magnification: Learning view synthesis using multiplane images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely · 2018
Earlier work this paper cites.
Digging into self-supervised monocular depth estimation
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow · 2019
Earlier work this paper cites.
Neural volumes: Learning dynamic renderable volumes from images
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh · 2019
Earlier work this paper cites.
Pushing the boundaries of view extrapolation with multiplane images
Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely · 2019
Earlier work this paper cites.
Neural point-based graphics
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky · 2020
Earlier work this paper cites.
Epipolar transformers
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu · 2020
Earlier work this paper cites.
Neural sparse voxel fields
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt · 2020
Earlier work this paper cites.
NeRF: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Earlier work this paper cites.
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger · 2020
Earlier work this paper cites.
Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun · 2020
Earlier work this paper cites.

Similar

Free view synthesis
Gernot Riegler and Vladlen Koltun · 2020
Cited alongside, same era.
Implicit neural representations with periodic activation functions
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein · 2020
Cited alongside, same era.
Single-view view synthesis with multiplane images
Richard Tucker and Noah Snavely · 2020
Cited alongside, same era.
Synsin: End-to-end view synthesis from a single image
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson · 2020
Cited alongside, same era.
Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan · 2021
Cited alongside, same era.
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Cited alongside, same era.
Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su · 2021
Cited alongside, same era.
Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes
Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll · 2021
Cited alongside, same era.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2021
Cited alongside, same era.
Point-based neural rendering with per-view optimization
Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis · 2021
Cited alongside, same era.
Infinite nature: Perpetual view generation of natural scenes from a single image
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa · 2021
Cited alongside, same era.
Nerf in the wild: Neural radiance fields for unconstrained photo collections
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth · 2021
Cited alongside, same era.

Then

Tensorf: Tensorial radiance fields
Original
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su · 2022
Later among the works it cites.
Plenoxels: Radiance fields without neural networks
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa · 2022
Later among the works it cites.
Fast and explicit neural view synthesis
Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M Susskind, and Qi Shan · 2022
Later among the works it cites.
Geonerf: Generalizing nerf with geometry priors
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret · 2022
Later among the works it cites.
Vision transformer for nerf-based view synthesis from a single input image
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi · 2022
Later among the works it cites.
Neural rays for occlusion-aware image-based rendering
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang · 2022
Later among the works it cites.
Instant neural graphics primitives with a multiresolution hash encoding
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller · 2022
Later among the works it cites.
Neural groundplans: Persistent neural scene representations from a single image
Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Andrei Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann · 2022
Later among the works it cites.
Volume rendering digest (for nerf)
Original
Andrea Tagliasacchi and Ben Mildenhall · 2022
Later among the works it cites.
Advances in neural rendering
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al · 2022
Later among the works it cites.
Neural fields in visual computing and beyond
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar · 2022
Later among the works it cites.
Video extrapolation in space and time
Yunzhi Zhang and Jiajun Wu · 2022
Later among the works it cites.
Generative novel view synthesis with 3d-aware diffusion models
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein · 2023
Closest in time.
Learning to render novel views from wide-baseline stereo pairs
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann · 2023
Closest in time.
3d gaussian splatting for real-time radiance field rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis · 2023
Closest in time.
Zeronvs: Zero-shot 360-degree view synthesis from a single real image
Original
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al · 2023
Closest in time.
Flowcam: Training generalizable 3d radiance fields without camera poses via pixel-aligned scene flow
Cameron Smith, Yilun Du, Ayush Tewari, and Vincent Sitzmann · 2023
Closest in time.
Viewset diffusion:(0-) image-conditioned 3d generative models from 2d data
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi · 2023
Closest in time.
Diffusion with forward models: Solving stochastic inverse problems without direct supervision
Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B Tenenbaum, Frédo Durand, William T Freeman, and Vincent Sitzmann · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…