2020

Accelerating 3D Deep Learning with PyTorch3D

Ravi, Nikhila, Reizenstein, Jeremy, Novotny, David et al.

Understand

Deep learning has significantly improved 2D image recognition.

Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition.
However despite growing interest, 3D deep learning remains relatively underexplored.
We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable.

Built on

Continuous shading of curved surfaces
H. Gouraud · 1971
Earlier work this paper cites.
The representation and matching of pictorial structures
M. A. Fischler and R. A. Elschlager · 1973
Earlier work this paper cites.
Illumination for computer generated pictures
B. T. Phong · 1975
Earlier work this paper cites.
The acronym model-based vision system
R. A. Brooks, R. Creiner, and T. O. Binford · 1979
Earlier work this paper cites.
Lectures in Pattern Theory I, II and III: Pattern Analysis, Pattern Synthesis and Regular Structures
U. Grenander · 1981
Earlier work this paper cites.
Model-based three-dimensional interpretations of two-dimensional images
R. A. Brooks · 1983
Earlier work this paper cites.
Fitting parameterized three-dimensional models to images
D. G. Lowe et al · 1991
Earlier work this paper cites.
Inverse rendering for computer graphics
S. R. Marschner and D. P. Greenberg · 1998
Earlier work this paper cites.
A taxonomy and evaluation of dense two-frame stereo correspondence algorithms
D. Scharstein and R. Szeliski · 2002
Earlier work this paper cites.
Face recognition based on fitting a 3d morphable model
V. Blanz and T. Vetter · 2003
Earlier work this paper cites.
Multiple view geometry in computer vision
R. Hartley and A. Zisserman · 2003
Earlier work this paper cites.
A survey of inverse rendering problems
G. Patow and X. Pueyo · 2003
Earlier work this paper cites.
High-performance software rasterization on gpus
S. Laine and T. Karras · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Depth map prediction from a single image using a multi-scale deep network
D. Eigen, C. Puhrsch, and R. Fergus · 2014
Earlier work this paper cites.
OpenDR: An approximate differentiable renderer
M. M. Loper and M. J. Black · 2014
Earlier work this paper cites.
Shapenet: An information-rich 3d model repository
Original
A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al · 2015
Earlier work this paper cites.
Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture
D. Eigen and R. Fergus · 2015
Earlier work this paper cites.
Fully convolutional networks for semantic segmentation
J. Long, E. Shelhamer, and T. Darrell · 2015
Earlier work this paper cites.
SMPL: A skinned multi-person linear model
M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black · 2015
Earlier work this paper cites.
Faster R-CNN: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Earlier work this paper cites.

Similar

ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Cited alongside, same era.
LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop
Original
F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al · 2016
Cited alongside, same era.
3D-R2N2: A unified approach for single and multi-view 3d object reconstruction
C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling
J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum · 2016
Cited alongside, same era.
A point set generation network for 3d object reconstruction from a single image
H. Fan, H. Su, and L. J. Guibas · 2017
Cited alongside, same era.
Mask R-CNN
K. He, G. Gkioxari, P. Dollár, and R. Girshick · 2017
Cited alongside, same era.
Billion-scale similarity search with gpus
Original
J. Johnson, M. Douze, and H. Jégou · 2017
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
T. N. Kipf and M. Welling · 2017
Cited alongside, same era.
Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs
M. Tatarchenko, A. Dosovitskiy, and T. Brox · 2017
Cited alongside, same era.

Then

Pixel2Mesh: Generating 3D mesh models from single RGB images
N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang · 2018
Later among the works it cites.
Learning to predict 3d objects with an interpolation-based differentiable renderer
W. Chen, H. Ling, J. Gao, E. Smith, J. Lehtinen, A. Jacobson, and S. Fidler · 2019
Later among the works it cites.
Fast graph representation learning with PyTorch Geometric
M. Fey and J. E. Lenssen · 2019
Later among the works it cites.
Mesh R-CNN
G. Gkioxari, J. Malik, and J. Johnson · 2019
Later among the works it cites.
Kaolin: A PyTorch library for accelerating 3d deep learning research
Original
K. M. Jatavallabhula, E. Smith, J.-F. Lafleche, C. F. Tsang, A. Rozantsev, W. Chen, and T. Xiang · 2019
Later among the works it cites.
Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction
S. Liu, W. Chen, T. Li, and H. Li · 2019
Later among the works it cites.
Occupancy networks: Learning 3d reconstruction in function space
L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger · 2019
Later among the works it cites.
Mitsuba 2: a retargetable forward and inverse renderer
M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob · 2019
Later among the works it cites.
C3dpo: Canonical 3d pose networks for non-rigid structure from motion
D. Novotny, N. Ravi, B. Graham, N. Neverova, and A. Vedaldi · 2019
Later among the works it cites.
Deepsdf: Learning continuous signed distance functions for shape representation
J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove · 2019
Later among the works it cites.
PyTorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al · 2019
Later among the works it cites.
PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization
S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li · 2019
Later among the works it cites.
Scene representation networks: Continuous 3d-structure-aware neural scene representations
V. Sitzmann, M. Zollhöfer, and G. Wetzstein · 2019
Later among the works it cites.
Geometrics: Exploiting geometric structure for graph-encoded objects
E. J. Smith, S. Fujimoto, A. Romero, and D. Meger · 2019
Later among the works it cites.
Tensorflow graphics: Computer graphics meets deep learning
J. Valentin, C. Keskin, P. Pidlypenskyi, A. Makadia, A. Sud, and S. Bouaziz · 2019
Later among the works it cites.
Pointflow: 3d point cloud generation with continuous normalizing flows
G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan · 2019
Later among the works it cites.
Differentiable surface splatting for point-based geometry processing
W. Yifan, F. Serena, S. Wu, C. Öztireli, and O. Sorkine-Hornung · 2019
Later among the works it cites.
Fast differentiable raycasting for neural rendering using sphere-based representations
Original
C. Lassner · 2020
Closest in time.
Nerf: Representing scenes as neural radiance fields for view synthesis
Original
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng · 2020
Closest in time.
Polygen: An autoregressive generative model of 3d meshes
Original
C. Nash, Y. Ganin, S. Eslami, and P. W. Battaglia · 2020
Closest in time.
SynSin: End-to-end view synthesis from a single image
O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…