2017

PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation

Xu, Danfei, Anguelov, Dragomir, Jain, Ashesh

Understand

We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information.

Unlike existing methods that either use multi-stage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic.
The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively.
The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors.

Built on

Histograms of oriented gradients for human detection
N. Dalal and B. Triggs · 2005
Earlier work this paper cites.
Simultaneous object recognition and segmentation from single or multiple model views
V. Ferrari, T. Tuytelaars, and L. Van Gool · 2006
Earlier work this paper cites.
3d object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints
F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce · 2006
Earlier work this paper cites.
The moped framework: Object recognition and pose estimation for manipulation
A. Collet, M. Martinez, and S. S. Srinivasa · 2011
Earlier work this paper cites.
A multilevel mixture-of-experts framework for pedestrian classification
M. Enzweiler and D. M. Gavrila · 2011
Earlier work this paper cites.
Are we ready for autonomous driving? the kitti vision benchmark suite
A. Geiger, P. Lenz, and R. Urtasun · 2012
Earlier work this paper cites.
Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models
M. Aubry, D. Maturana, A. A. Efros, B. C. Russell, and J. Sivic · 2014
Earlier work this paper cites.
A multi-sensor fusion system for moving object detection and tracking in urban driving environments
H. Cho, Y.-W. Seo, B. V. Kumar, and R. R. Rajkumar · 2014
Earlier work this paper cites.
Scalable object detection using deep neural networks
D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Sliding shapes for 3d object detection in depth images
S. Song and J. Xiao · 2014
Earlier work this paper cites.
Single image 3d object detection and pose estimation for grasping
M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis · 2014
Earlier work this paper cites.
3d object proposals for accurate object class detection
X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and R. Urtasun · 2015
Earlier work this paper cites.

Similar

Multi-modal sensor registration for vehicle perception via deep neural networks
M. Giering, V. Venugopalan, and K. Reddy · 2015
Cited alongside, same era.
Aligning 3d models to rgb-d images of cluttered scenes
S. Gupta, P. Arbeláez, R. Girshick, and J. Malik · 2015
Cited alongside, same era.
Densebox: Unifying landmark localization with end to end object detection
Original
L. Huang, Y. Yang, Y. Deng, and Y. Yu · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Cited alongside, same era.
Sun rgb-d: A rgb-d scene understanding benchmark suite
S. Song, S. P. Lichtenberg, and J. Xiao · 2015
Cited alongside, same era.
Viewpoints and keypoints
S. Tulsiani and J. Malik · 2015
Cited alongside, same era.
Data-driven 3d voxel patterns for object category recognition
Y. Xiang, W. Choi, Y. Lin, and S. Savarese · 2015
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
3d fully convolutional network for vehicle detection in point cloud
B. Li · 2016
Cited alongside, same era.
Vehicle detection from 3d lidar using fully convolutional network
Original
B. Li, T. Zhang, and T. Xia · 2016
Cited alongside, same era.
Pointnet: Deep learning on point sets for 3d classification and segmentation
Original
C. R. Qi, H. Su, K. Mo, and L. J. Guibas · 2016
Cited alongside, same era.

Then

You only look once: Unified, real-time object detection
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi · 2016
Later among the works it cites.
Three-dimensional object detection and layout prediction using clouds of oriented gradients
Z. Ren and E. B. Sudderth · 2016
Later among the works it cites.
Deep sliding shapes for amodal 3d object detection in rgb-d images
S. Song and J. Xiao · 2016
Later among the works it cites.
Multi-view 3d object detection network for autonomous driving
X. Chen, H. Ma, J. Wan, B. Li, and T. Xia · 2017
Closest in time.
Speed/accuracy trade-offs for modern convolutional object detectors
J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy · 2017
Closest in time.
2d-driven 3d object detection in rgb-d images
J. Lahoud and B. Ghanem · 2017
Closest in time.
Focal loss for dense object detection
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár · 2017
Closest in time.
3d bounding box estimation using deep learning and geometry
A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka · 2017
Closest in time.
Feature pyramid networks for object detection
T.-Y. L. nad Piotr Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie · 2017
Closest in time.
Octnet: Learning deep representations at high resolution
G. Riedler, A. O. Ulusoy, and A. Geiger · 2017
Closest in time.
Sparsity invariant cnns
Original
J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger · 2017
Closest in time.
Subcategory-aware convolutional neural networks for object proposals and detection
Y. Xiang, W. Choi, Y. Lin, and S. Savarese · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…