2019

SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization

Jiang, Yue, Ji, Dantong, Han, Zhizhong et al.

Understand

We propose SDFDiff, a novel approach for image-based shape optimization using differentiable rendering of 3D shapes represented by signed distance functions (SDFs).

Compared to other representations, SDFs have the advantage that they can represent shapes with arbitrary topology, and that they guarantee watertight surfaces.
We apply our approach to the problem of multi-view 3D reconstruction, where we achieve high reconstruction quality and can capture complex topology of 3D objects.
In addition, we employ a multi-resolution strategy to obtain a robust optimization algorithm.

Built on

A volumetric method for building complex models from range images
Brian Curless and Marc Levoy · 1996
Earlier work this paper cites.
Sphere Tracing: a geometric method for the antialiased ray tracing of implicit surfaces
John C. Hart · 1996
Earlier work this paper cites.
A PDE-Based Fast Local Level Set Method
Danping Peng, Barry Merriman, Stanley Osher, Hongkai Zhao, and Myungjoo Kang · 1999
Earlier work this paper cites.
A signal-processing framework for inverse rendering
Ravi Ramamoorthi and Pat Hanrahan · 2001
Earlier work this paper cites.
Surface splatting
Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross · 2001
Earlier work this paper cites.
Level Set Methods and Dynamic Implicit Surfaces
S Osher, R Fedkiw, and K Piechor · 2004
Earlier work this paper cites.
A comparison and evaluation of multi-view stereo reconstruction algorithms
S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski · 2006
Earlier work this paper cites.
Gradient flows for optimizing triangular mesh-based surfaces: Applications to 3D reconstruction problems dealing with visibility
Amaël Delaunoy and Emmanuel Prados · 2011
Earlier work this paper cites.
KinectFusion: Real-time dense surface mapping and tracking
Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon · 2011
Earlier work this paper cites.
Real-time 3D reconstruction at scale using voxel hashing
Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
OpenDR: An approximate differentiable renderer
Matthew M. Loper and Michael J. Black · 2014
Earlier work this paper cites.
A new variational framework for multiview surface reconstruction
Ben Semerjian · 2014
Earlier work this paper cites.
ShapeNet: An information-rich 3D model repository
Original
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu · 2015
Earlier work this paper cites.
3D-R2N2: A unified approach for single and multi-view 3D object reconstruction
Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese · 2016
Earlier work this paper cites.
3D shape induction from 2D views of multiple objects
Matheus Gadelha, Subhransu Maji, and Rui Wang · 2016
Earlier work this paper cites.
Unsupervised learning of 3D structure from images
Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess · 2016
Earlier work this paper cites.
Shading-aware multi-view stereo
F. Langguth, K. Sunkavalli, S. Hadap, and M. Goesele · 2016
Earlier work this paper cites.
Perspective Transformer Nets: Learning single-view 3D object reconstruction without 3D supervision
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee · 2016
Earlier work this paper cites.

Similar

Representation learning and adversarial generation of 3D point clouds
Original
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J. Guibas · 2017
Cited alongside, same era.
A point set generation network for 3D object reconstruction from a single image
Haoqiang Fan, Hao Su, and Leonidas J. Guibas · 2017
Cited alongside, same era.
Weakly supervised 3D reconstruction with adversarial constraint
J. Gwak, C. B. Choy, M. Chandraker, A. Garg, and S. Savarese · 2017
Cited alongside, same era.
Refining multi-view 3d reconstruction with differentiable rendering, 2017
Simone Raimondi · 2017
Cited alongside, same era.
Learning detailed face reconstruction from a single image
Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel · 2017
Cited alongside, same era.
Learning category-specific deformable 3D models for object reconstruction
S. Tulsiani, A. Kar, J. Carreira, and J. Malik · 2017
Cited alongside, same era.
Multi-view supervision for single-view reconstruction via differentiable ray consistency
Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik · 2017
Cited alongside, same era.
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum · 2017
Cited alongside, same era.
Synthesizing robust adversarial examples
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok · 2018
Cited alongside, same era.
Inverse Transport Networks
Original
Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas · 2018
Cited alongside, same era.
Learning to generate and reconstruct 3D meshes with only 2D supervision
Paul Henderson and Vittorio Ferrari · 2018
Cited alongside, same era.
Unsupervised learning of shape and pose with differentiable point clouds
Eldar Insafutdinov and Alexey Dosovitskiy · 2018
Cited alongside, same era.

Then

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yong-Liang Yang · 2018
Later among the works it cites.
End-to-end 6-DoF object pose estimation through differentiable rasterization
Andrea Palazzi, Luca Bergamini, Simone Calderara, and Rita Cucchiara · 2018
Later among the works it cites.
First experiments with neural translation of informal to formal mathematics
Original
Qingxiang Wang, Cezary Kaliszyk, and Josef Urban · 2018
Later among the works it cites.
Visual Object Networks: Image generation with disentangled 3D representations
Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, and William T. Freeman · 2018
Later among the works it cites.
Learning to predict 3d objects with an interpolation-based differentiable renderer
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler · 2019
Closest in time.
Learning implicit fields for generative shape modeling
Zhiqin Chen and Hao Zhang · 2019
Closest in time.
View Inter-Prediction GAN: Unsupervised representation learning for 3D shapes by learning global shape memories to support local view predictions
Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker · 2019
Closest in time.
Y2Seq2Seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences
Zhizhong Han, Mingyang Shang, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker · 2019
Closest in time.
Escaping Plato’s Cave using adversarial training: 3d shape from unstructured 2d image collections
P Henzler, N Mitra, and T Ritschel · 2019
Closest in time.
Soft Rasterizer: A differentiable renderer for image-based 3D reasoning
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li · 2019
Closest in time.
Occupancy networks: Learning 3D reconstruction in function space
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger · 2019
Closest in time.
Deep Level Sets: Implicit surface representations for 3D shape inference
Original
Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders P. Eriksson · 2019
Closest in time.
CAPNet: Continuous approximation projection for 3D point cloud reconstruction using 2d supervision
KL Navaneet, Priyanka Mandikal, Mayank Agarwal, and R Venkatesh Babu · 2019
Closest in time.
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Original
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove · 2019
Closest in time.
Pix2Vex: Image-to-geometry reconstruction using a smooth differentiable renderer
Original
Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or · 2019
Closest in time.
Scene Representation Networks: Continuous 3D-structure-aware neural scene representations
Original
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein · 2019
Closest in time.
Differentiable surface splatting for point-based geometry processing
Yifan Wang, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung · 2019
Closest in time.
DIST: Rendering deep implicit signed distance function with differentiable sphere tracing
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui · 2020
Closest in time.
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…