2020

Stable View Synthesis

Riegler, Gernot, Koltun, Vladlen

Understand

We present Stable View Synthesis (SVS).

Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene.
The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo.
Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images.

Built on

A Space-Sweep Approach to True Multi-Image Matching
Robert T Collins · 1996
Earlier work this paper cites.
Review of image-based rendering techniques
Harry Shum and Sing Bing Kang · 2000
Earlier work this paper cites.
Surface Light Fields for 3D Photography
Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle · 2000
Earlier work this paper cites.
Unstructured Lumigraph Rendering
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen · 2001
Earlier work this paper cites.
Image Quality Assessment: From Error Measurement to Structural Similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli · 2004
Earlier work this paper cites.
High-quality video view interpolation using a layered representation
C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski · 2004
Earlier work this paper cites.
Unstructured Light Fields
Abe Davis, Marc Levoy, and Fredo Durand · 2012
Earlier work this paper cites.
Depth Synthesis and Local Warps for Plausible Image-based Navigation
Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis · 2013
Earlier work this paper cites.
First-person Hyper-lapse Videos
Johannes Kopf, Michael F Cohen, and Richard Szeliski · 2014
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
U-Net: Convolutional Networks for Biomedical Image Segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox · 2015
Earlier work this paper cites.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan and Andrew Zisserman · 2015
Earlier work this paper cites.
Large-Scale Data for Multiple-View Stereopsis
Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl · 2016
Earlier work this paper cites.
DeepStereo: Learning to Predict New Views from the World’s Imagery
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely · 2016
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Scalable inside-out image-based rendering
Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow · 2016
Earlier work this paper cites.

Similar

Learning-Based View Synthesis for Light Field Cameras
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi · 2016
Cited alongside, same era.
Structure-from-Motion Revisited
Johannes Lutz Schönberger and Jan-Michael Frahm · 2016
Cited alongside, same era.
Pixelwise View Selection for Unstructured Multi-View Stereo
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys · 2016
Cited alongside, same era.
Photographic Image Synthesis with Cascaded Refinement Networks
Qifeng Chen and Vladlen Koltun · 2017
Cited alongside, same era.
Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun · 2017
Cited alongside, same era.
Soft 3D Reconstruction for View Synthesis
Eric Penner and Li Zhang · 2017
Cited alongside, same era.
PointNet: Deep learning on Point Sets for 3D Classification and Segmentation
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas · 2017
Cited alongside, same era.
Deep Blending for Free-Viewpoint Image-Based Rendering
Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow · 2018
Cited alongside, same era.
Graph Attention Networks
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio · 2018
Cited alongside, same era.
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Cited alongside, same era.
Stereo Magnification: Learning view synthesis using multiplane images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely · 2018
Cited alongside, same era.
Extreme View Synthesis
Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz · 2019
Cited alongside, same era.

Then

PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Later among the works it cites.
DeepVoxels: Learning Persistent 3D Feature Embeddings
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer · 2019
Later among the works it cites.
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein · 2019
Later among the works it cites.
Pushing the Boundaries of View Extrapolation with Multiplane Images
Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely · 2019
Later among the works it cites.
Deferred Neural Rendering: Image Synthesis using Neural Textures
Justus Thies, Michael Zollhöfer, and Matthias Nießner · 2019
Later among the works it cites.
Deep View Synthesis from Sparse Photometric Images
Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi · 2019
Later among the works it cites.
Neural Point-Based Graphics
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky · 2020
Closest in time.
Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi · 2020
Closest in time.
Neural Point Cloud Rendering via Multi-Plane Projection
Peng Dai, Yinda Zhang, Zhuwen Li, Shuaicheng Liu, and Bing Zeng · 2020
Closest in time.
Crowdsampling the Plenoptic Function
Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely · 2020
Closest in time.
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections
Original
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth · 2020
Closest in time.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2020
Closest in time.
Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger · 2020
Closest in time.
Free View Synthesis
Gernot Riegler and Vladlen Koltun · 2020
Closest in time.
Image-guided Neural Object Rendering
Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner · 2020
Closest in time.
NeRF++: Analyzing and Improving Neural Radiance Fields
Original
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…