2019

Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation

Jeong, Rae, Aytar, Yusuf, Khosid, David et al.

Understand

Collecting and automatically obtaining reward signals from real robotic visual data for the purposes of training reinforcement learning algorithms can be quite challenging and time-consuming.

Methods for utilizing unlabeled data can have a huge potential to further accelerate robotic learning.
We consider here the problem of performing manipulation tasks from pixels.
In such tasks, choosing an appropriate state representation is crucial for planning and control.

Built on

V.-D. Nguyen, “Constructing Force-Closure Grasps,” IJRR , 1988
1988
Earlier work this paper cites.
S. Chopra, R. Hadsell, Y. LeCun, et al. , “Learning a similarity metric discriminatively, with application to face verification,” in CVPR (1) , 2005, pp. 539–546
2005
Earlier work this paper cites.
R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) , vol. 2. IEEE, 2006, pp. 1735–1742
2006
Earlier work this paper cites.
B. Siciliano and O. Khatib, Springer Handbook of Robotics . Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2007
2007
Earlier work this paper cites.
A. Rodriguez, M. T. Mason, and S. Ferry, “From caging to grasping,” IJRR , 2012
2012
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on . IEEE, 2012, pp. 5026–5033
2012
Earlier work this paper cites.
L. Grne and J. Pannek, Nonlinear Model Predictive Control: Theory and Algorithms . Springer Publishing Company, Incorporated, 2013
2013
Earlier work this paper cites.
J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp synthesis-a survey,” IEEE Transactions on Robotics , 2014
2014
Earlier work this paper cites.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-Adversarial Training of Neural Networks,” arXiv e-prints , May 2015
2015
Earlier work this paper cites.
S. Kolev and E. Todorov, “Physically consistent state estimation and system identification for contacts,” in 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) , Nov 2015, pp. 1036–1043
2015
Earlier work this paper cites.
R. Caseiro, J. F. Henriques, P. Martins, and J. Batista, “Beyond the shortest path: Unsupervised Domain Adaptation by Sampling Subspaces Along the Spline Flow,” in CVPR , 2015
2015
Earlier work this paper cites.
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” CoRR , vol. abs/1603.02199, 2016
Original
2016
Earlier work this paper cites.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” The Journal of Machine Learning Research , vol. 17, no. 1, pp. 2096–2030, 2016
2016
Earlier work this paper cites.
K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, “Domain separation networks,” in NIPS , 2016
2016
Earlier work this paper cites.
I. Misra, C. L. Zitnick, and M. Hebert, “Shuffle and learn: unsupervised learning using temporal order verification,” in European Conference on Computer Vision . Springer, 2016, pp. 527–544
2016
Earlier work this paper cites.

Similar

K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in Advances in Neural Information Processing Systems , 2016, pp. 1857–1865
2016
Cited alongside, same era.
J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences of the United States of America , vol. 114 13, pp. 3521–3526, 2016
2016
Cited alongside, same era.
U. Viereck, A. t. Pas, K. Saenko, and R. Platt, “Learning a visuomotor controller for real world robotic grasping using easily simulated depth images,” CoRL , 2017
2017
Cited alongside, same era.
J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” in RSS , 2017
2017
Cited alongside, same era.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” CoRR , vol. abs/1710.06537, 2017
Original
2017
Cited alongside, same era.
K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke, “Using simulation and domain adaptation to improve efficiency of deep robotic grasping,” CoRR , vol. abs/1709.07857, 2017
Original
2017
Cited alongside, same era.
G. Csurka, “Domain adaptation for visual applications: A comprehensive survey,” arxiv:1702.05374 , 2017
Original
2017
Cited alongside, same era.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” CoRR , vol. abs/1703.06907, 2017
Original
2017
Cited alongside, same era.
S. Makita and W. Wan, “A Survey of Robotic Caging and its Applications,” Advanced Robotics , vol. 0, no. 0, pp. 1–15, 2017
2017
Cited alongside, same era.
K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised pixel-level domain adaptation with generative adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 3722–3731
2017
Cited alongside, same era.
B. Fernando, H. Bilen, E. Gavves, and S. Gould, “Self-supervised video representation learning with odd-one-out networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 3636–3645
2017
Cited alongside, same era.
P. Sermanet, C. Lynch, J. Hsu, and S. Levine, “Time-contrastive networks: Self-supervised learning from multi-view observation,” CoRR , vol. abs/1704.06888, 2017
Original
2017
Cited alongside, same era.

Then

F. Sadeghi and S. Levine, “CAD2RL: Real single-image flight without a single real image.” in RSS , 2017
2017
Later among the works it cites.
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” CoRR , vol. abs/1806.10293, 2018
Original
2018
Later among the works it cites.
S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis, “Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks,” CoRR , vol. abs/1812.07252, 2018
Original
2018
Later among the works it cites.
OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning dexterous in-hand manipulation,” CoRR , vol. abs/1808.00177, 2018
Original
2018
Later among the works it cites.
G. J. Stein and N. Roy, “Genesis-rt: Generating synthetic images for training secondary real-world tasks,” in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 7151–7158
2018
Later among the works it cites.
Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” CoRR , vol. abs/1810.05687, 2018
Original
2018
Later among the works it cites.
M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg, “Learning by playing solving sparse reward tasks from scratch,” in International Conference on Machine Learning , 2018, pp. 4341–4350
2018
Later among the works it cites.
Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess, “Reinforcement and imitation learning for diverse visuomotor skills,” in Proceedings of Robotics: Science and Systems , Pittsburgh, Pennsylvania, June 2018
2018
Later among the works it cites.
D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, “Learning and using the arrow of time,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 8052–8060
2018
Later among the works it cites.
Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas, “Playing hard exploration games by watching youtube,” in Advances in Neural Information Processing Systems , 2018, pp. 2930–2941
2018
Later among the works it cites.
A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. A. Riedmiller, “Maximum a posteriori policy optimisation,” CoRR , vol. abs/1806.06920, 2018
Original
2018
Later among the works it cites.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018
Original
2018
Later among the works it cites.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller, “Deepmind control suite,” ArXiv , vol. abs/1801.00690, 2018
Original
2018
Later among the works it cites.
A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” Science , vol. 364, no. 6446, p. eaat8414, 2019
2019
Closest in time.
D. Schwab, J. T. Springenberg, M. F. Martins, T. Lampe, M. Neunert, A. Abdolmaleki, T. Hertweck, R. Hafner, F. Nori, and M. A. Riedmiller, “Simultaneously learning vision and feature-based control policies for real-world ball-in-a-cup,” CoRR , vol. abs/1902.04706, 2019
Original
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…