2020

Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras

Akinola, Iretiayo, Varley, Jacob, Kalashnikov, Dmitry

Understand

In this work, we present an effective multi-view approach to closed-loop end-to-end learning of precise manipulation tasks that are 3D in nature.

Our method learns to accomplish these tasks using multiple statically placed but uncalibrated RGB camera views without building an explicit 3D representation such as a pointcloud or voxel grid.
This multi-camera approach achieves superior task performance on difficult stacking and insertion tasks compared to single-view baselines.
Single view robotic agents struggle from occlusion and challenges in estimating relative poses between points of interest.

Built on

C. Papazov and D. Burschka, “An efficient ransac for 3d object recognition in noisy and occluded scenes,” in Asian Conference on Computer Vision . Springer, 2010, pp. 135–148
2010
Earlier work this paper cites.
M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal, “Learning force control policies for compliant manipulation,” in 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2011, pp. 4639–4644
2011
Earlier work this paper cites.
F. Lange, M. Suppa, and G. Hirzinger, “Control with a compliant force-torque sensor,” in ROBOTIK 2012; 7th German Conference on Robotics . VDE, 2012, pp. 1–6
2012
Earlier work this paper cites.
T. Schmidt, R. A. Newcombe, and D. Fox, “Dart: Dense articulated real-time tracking.” in Robotics: Science and Systems , vol. 2, no. 1. Berkeley, CA, 2014
2014
Earlier work this paper cites.
J. Mišeikis, K. Glette, O. J. Elle, and J. Torresen, “Multi 3d camera mapping for predictive and reflexive robot manipulator trajectory estimation,” in 2016 IEEE Symposium Series on Computational Intelligence (SSCI) . IEEE, 2016, pp. 1–8
2016
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” GitHub , 2016
2016
Earlier work this paper cites.
I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, and M. Riedmiller, “Data-efficient deep reinforcement learning for dexterous manipulation,” arXiv preprint arXiv:1704.03073 , 2017
Original
2017
Earlier work this paper cites.
M. Večerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller, “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards,” arXiv preprint arXiv:1707.08817 , 2017
Original
2017
Earlier work this paper cites.
J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen, “Shape completion enabled robotic grasping,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 2442–2447
2017
Earlier work this paper cites.
A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,” The International Journal of Robotics Research , vol. 36, no. 13-14, pp. 1455–1473, 2017
2017
Earlier work this paper cites.
G. Izatt, G. Mirano, E. Adelson, and R. Tedrake, “Tracking objects with point clouds from vision and touch,” in 2017 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2017, pp. 4000–4007
2017
Earlier work this paper cites.
M. Gualtieri and R. Platt, “Viewpoint selection for grasp detection,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2017, pp. 258–264
2017
Earlier work this paper cites.
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. , “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” Conference on Robot Learning (CoRL) , 2018
2018
Earlier work this paper cites.

Similar

D. Morrison, P. Corke, and J. Leitner, “Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach,” in Proc. of Robotics: Science and Systems (RSS) , 2018
2018
Cited alongside, same era.
A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. , “Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 1–8
2018
Cited alongside, same era.
A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, “Learning synergies between pushing and grasping with self-supervised deep reinforcement learning,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 4238–4245
2018
Cited alongside, same era.
A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Overcoming exploration in reinforcement learning with demonstrations,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 6292–6299
2018
Cited alongside, same era.
G. Thomas, M. Chien, A. Tamar, J. A. Ojea, and P. Abbeel, “Learning robotic assembly from cad,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 1–9
2018
Cited alongside, same era.
X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee, “Learning 6-dof grasping interaction via deep geometry-aware 3d representations,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 1–9
2018
Cited alongside, same era.
J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” arXiv preprint arXiv:1809.10790 , 2018
Original
2018
Cited alongside, same era.
M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. , “Learning dexterous in-hand manipulation,” arXiv preprint arXiv:1808.00177 , 2018
Original
2018
Cited alongside, same era.
S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al. , “Neural scene representation and rendering,” Science , vol. 360, no. 6394, pp. 1204–1210, 2018
2018
Cited alongside, same era.
P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 1134–1141
2018
Cited alongside, same era.
R. Cheng, A. Agarwal, and K. Fragkiadaki, “Reinforcement learning of active vision formanipulating objects under occlusions,” arXiv preprint arXiv:1811.08067 , 2018
Original
2018
Cited alongside, same era.
S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” The International Journal of Robotics Research , vol. 37, no. 4-5, pp. 421–436, 2018
2018
Cited alongside, same era.

Then

B. Wu, I. Akinola, and P. K. Allen, “Pixel-attentive policy gradient for multi-fingered grasping in cluttered scenes,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2019
2019
Later among the works it cites.
O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” arXiv preprint arXiv:1907.03146 , 2019
Original
2019
Later among the works it cites.
M. Vecerik, O. Sushkov, D. Barker, T. Rothörl, T. Hester, and J. Scholz, “A practical approach to insertion with variable socket position using deep reinforcement learning,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 754–760
2019
Later among the works it cites.
X. Chen, R. Chen, Z. Sui, Z. Ye, Y. Liu, R. Bahar, and O. C. Jenkins, “Grip: Generative robust inference and perception for semantic robot manipulation in adversarial environments,” arXiv preprint arXiv:1903.08352 , 2019
Original
2019
Later among the works it cites.
Y. Litvak, A. Biess, and A. Bar-Hillel, “Learning pose estimation for high-precision robotic assembly using simulated depth images,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 3521–3527
2019
Later among the works it cites.
L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “kpam: Keypoint affordances for category-level robotic manipulation,” arXiv preprint arXiv:1903.06684 , 2019
Original
2019
Later among the works it cites.
J. Luo, E. Solowjow, C. Wen, J. A. Ojea, A. M. Agogino, A. Tamar, and P. Abbeel, “Reinforcement learning on variable impedance controller for high-precision robotic assembly,” arXiv preprint arXiv:1903.01066 , 2019
Original
2019
Later among the works it cites.
D. Watkins-Valls, J. Varley, and P. Allen, “Multi-modal geometric learning for grasping and manipulation,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 7339–7345
2019
Later among the works it cites.
M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,” arXiv preprint arXiv:1907.13098 , 2019
Original
2019
Later among the works it cites.
D. Jain, A. Li, S. Singhal, A. Rajeswaran, V. Kumar, and E. Todorov, “Learning deep visuomotor policies for dexterous hand manipulation,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 3636–3643
2019
Later among the works it cites.
S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis, “Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019, pp. 12 627–12 637
2019
Later among the works it cites.
Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 8973–8979
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…