2017

Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task

James, Stephen, Davison, Andrew J., Johns, Edward

Understand

End-to-end control for robot manipulation and grasping is emerging as an attractive alternative to traditional pipelined approaches.

However, end-to-end methods tend to either be slow to train, exhibit little or no generalisability, or lack the ability to accomplish long-horizon or multi-stage tasks.
In this paper, we show how two simple techniques can lead to end-to-end (image to velocity) execution of a multi-stage task, which is analogous to a simple tidying routine, without having seen a single real image.
This involves locating, reaching for, and grasping a cube, then locating a basket and dropping the cube inside.

Built on

Improving noise
K. Perlin · 2002
Earlier work this paper cites.
Guided policy search
S. Levine and V. Koltun · 2013
Earlier work this paper cites.
V-rep: a versatile and scalable robot simulation framework
M. F. E. Rohmer, S. P. N. Singh · 2013
Earlier work this paper cites.
Learning neural network policies with guided policy search under unknown dynamics
S. Levine and P. Abbeel · 2014
Earlier work this paper cites.
Towards vision-based deep reinforcement learning for robotic motion control
F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al · 2015
Earlier work this paper cites.
Learning from multiple demonstrations using trajectory-aware non-rigid registration with applications to deformable object manipulation
A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. Kingma and J. Ba · 2015
Earlier work this paper cites.
End-to-end training of deep visuomotor policies
S. Levine, C. Finn, T. Darrell, and P. Abbeel · 2016
Earlier work this paper cites.
Guided policy search via approximate mirror descent
W. H. Montgomery and S. Levine · 2016
Earlier work this paper cites.
Reset-free guided policy search: Efficient deep reinforcement learning with stochastic initial states
W. Montgomery, A. Ajay, C. Finn, P. Abbeel, and S. Levine · 2016
Earlier work this paper cites.

Similar

3d simulation for robot arm control with deep q-learning
S. James and E. Johns · 2016
Cited alongside, same era.
Reinforcement learning with unsupervised auxiliary tasks
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu · 2016
Cited alongside, same era.
(cad)2rl: Real single-image flight without a single real image
F. Sadeghi and S. Levine · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al · 2016
Cited alongside, same era.
Adapting deep visuomotor representations with weak pairwise constraints
E. Tzeng, C. Devin, J. Hoffman, C. Finn, P. Abbeel, S. Levine, K. Saenko, and T. Darrell · 2016
Cited alongside, same era.
Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours
L. Pinto and A. Gupta · 2016
Cited alongside, same era.
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen · 2016
Cited alongside, same era.
Deep learning a grasp function for grasping under gripper pose uncertainty
E. Johns, S. Leutenegger, and A. J. Davison · 2016
Cited alongside, same era.
Transfer from simulation to real world through learning deep inverse dynamics model
Original
P. Christiano, Z. Shah, I. Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba · 2016
Cited alongside, same era.
Data-efficient deep reinforcement learning for dexterous manipulation
Original
I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, and M. Riedmiller · 2017
Cited alongside, same era.
Feature control as intrinsic motivation for hierarchical reinforcement learning
Original
N. Dilokthanakul, C. Kaplanis, N. Pawlowski, and M. Shanahan · 2017
Cited alongside, same era.

Then

Domain randomization for transferring deep neural networks from simulation to the real world
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel · 2017
Closest in time.
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
S. Gu, E. Holly, T. Lillicrap, and S. Levine · 2017
Closest in time.
Sim-to-real robot learning from pixels with progressive nets
A. A. Rusu, M. Vecerik, T. Rothorl, N. Heess, R. Pascanu, and R. Hadsell · 2017
Closest in time.
One-shot imitation learning
Original
Y. Duan, M. Andrychowicz, B. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba · 2017
Closest in time.
Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets
K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. Lim · 2017
Closest in time.
Third-person imitation learning
B. C. Stadie, P. Abbeel, and I. Sutskever · 2017
Closest in time.
Deep visual foresight for planning robot motion
C. Finn and S. Levine · 2017
Closest in time.
Learning modular neural network policies for multi-task and multi-robot transfer
C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine · 2017
Closest in time.
Learning invariant feature spaces to transfer skills with reinforcement learning
A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine · 2017
Closest in time.
Darla: Improving zero-shot transfer in reinforcement learning
I. Higgins, A. Pal, A. A. Rusu, L. Matthey, C. P. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…