2018

Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

Tremblay, Jonathan, To, Thang, Sundaralingam, Balakumar et al.

Understand

Using synthetic data for training deep neural networks for robotic manipulation holds the promise of an almost unlimited amount of pre-labeled training data, generated safely out of harm's way.

One of the key challenges of synthetic data, to date, has been to bridge the so-called reality gap, so that networks trained on synthetic data operate correctly when exposed to real-world data.
We explore the reality gap in the context of 6-DoF pose estimation of known objects from a single RGB image.
We show that for this problem the reality gap can be successfully spanned by a simple combination of domain randomized and photorealistic data.

Built on

EP n P: An accurate O( n ) solution to the P n P problem
V. Lepetit, F. Moreno-Noguer, and P. Fua · 2009
Earlier work this paper cites.
Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes
S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab · 2012
Earlier work this paper cites.
Gradient response maps for real-time detection of textureless objects
S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, P. Fua, and V. Lepetit · 2012
Earlier work this paper cites.
A naturalistic open source movie for optical flow evaluation
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black · 2012
Earlier work this paper cites.
Microsoft COCO: Common objects in context
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Learning 6D object pose estimation using 3D object coordinates
E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Earlier work this paper cites.
The YCB object and model set: Towards common benchmarks for manipulation research
B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba · 2015
Earlier work this paper cites.
Comparing model predictive control and input shaping for improved response of low-impedance robots
L. Rupert, P. Hyatt, and M. D. Killpack · 2015
Earlier work this paper cites.
SimTrack: A simulation-based framework for scalable real-time object pose detection and tracking
K. Pauwels and D. Kragic · 2015
Earlier work this paper cites.
SceneNet: Understanding real world indoor scenes with synthetic data
A. Handa, V. Pătrăucean, V. Badrinarayanan, S. Stent, and R. Cipolla · 2015
Earlier work this paper cites.
FlowNet: Learning optical flow with convolutional networks
A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. Smagt, D. Cremers, and T. Brox · 2015
Earlier work this paper cites.
R-FCN: Object detection via region-based fully convolutional networks
J. Dai, Y. Li, K. He, and J. Sun · 2016
Earlier work this paper cites.
SSD: Single shot multibox detector
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg · 2016
Earlier work this paper cites.
Convolutional pose machines
S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh · 2016
Earlier work this paper cites.

Similar

Real-time pose detection and tracking of hundreds of objects
K. Pauwels, L. Rubio, and E. Ros · 2016
Cited alongside, same era.
A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation
N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox · 2016
Cited alongside, same era.
UnrealCV: Connecting computer vision to unreal engine
W. Qiu and A. Yuille · 2016
Cited alongside, same era.
UnrealStereo: A synthetic dataset for analyzing stereo vision
Original
Y. Zhang, W. Qiu, Q. Chen, X. Hu, and A. Yuille · 2016
Cited alongside, same era.
The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes
G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. Lopez · 2016
Cited alongside, same era.
Playing for data: Ground truth from computer games
S. R. Richter, V. Vineet, S. Roth, and V. Koltun · 2016
Cited alongside, same era.
Virtual worlds as proxy for multi-object tracking analysis
A. Gaidon, Q. Wang, Y. Cabon, and E. Vig · 2016
Cited alongside, same era.
YOLO9000: Better, faster, stronger
J. Redmon and A. Farhadi · 2017
Cited alongside, same era.
BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth
M. Rad and V. Lepetit · 2017
Cited alongside, same era.
Domain randomization for transferring deep neural networks from simulation to the real world
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel · 2017
Cited alongside, same era.
CAD 2 RL: Real single-image flight without a single real image
F. Sadeghi and S. Levine · 2017
Cited alongside, same era.
Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks
F. Zhang, J. Leitner, M. Milford, and P. Corke · 2017
Cited alongside, same era.

Then

End-to-end learning of semantic grasping
E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz, and S. Levine · 2017
Later among the works it cites.
Real-time monocular pose estimation of 3D objects using temporally consistent local color histograms
H. Tjaden, U. Schwanecke, and E. Schömer · 2017
Later among the works it cites.
3D bounding box estimation using deep learning and geometry
A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka · 2017
Later among the works it cites.
6-DoF object pose from semantic keypoints
G. Pavlakos, X. Zhou, A. Chan, K. G. Derpanis, and K. Daniilidis · 2017
Later among the works it cites.
A self-supervised learning system for object detection using physics simulation and multi-view pose estimation
C. Mitash, K. E. Bekris, and A. Boularias · 2017
Later among the works it cites.
SceneNet RGB-D: 5M photorealistic images of synthetic indoor trajectories with ground truth
J. McCormac, A. Handa, and S. Leutenegger · 2017
Later among the works it cites.
Procedural modeling and physically based rendering for synthetic data generation in automotive applications
A. Tsirikoglou, J. Kronander, M. Wrenninge, and J. Unger · 2017
Later among the works it cites.
PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes
Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox · 2018
Closest in time.
Real-time seamless single shot 6D object pose prediction
B. Tekin, S. N. Sinha, and P. Fua · 2018
Closest in time.
Falling things: A synthetic dataset for 3D object detection and pose estimation
J. Tremblay, T. To, and S. Birchfield · 2018
Closest in time.
LabelFusion: A pipeline for generating ground truth labels for real RGBD data of cluttered scenes
P. Marion, P. R. Florence, L. Manuelli, and R. Tedrake · 2018
Closest in time.
NDDS: NVIDIA deep learning dataset synthesizer, 2018
T. To, J. Tremblay, D. McKay, Y. Yamaguchi, K. Leung, A. Balanon, J. Cheng, and S. Birchfield · 2018
Closest in time.
Sim4CV: A photo-realistic simulator for computer vision applications
M. Mueller, V. Casser, J. Lahoud, N. Smith, and B. Ghanem · 2018
Closest in time.
Training deep networks with synthetic data: Bridging the reality gap by domain randomization
J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield · 2018
Closest in time.
Implicit 3D orientation learning for 6D object detection from RGB images
M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel · 2018
Closest in time.
Discovery of latent 3D keypoints via end-to-end geometric reasoning
S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…