2020

Scalable Multi-Task Imitation Learning with Autonomous Improvement

Singh, Avi, Jang, Eric, Irpan, Alexander et al.

Understand

While robot learning has demonstrated promising results for enabling robots to automatically acquire new skills, a critical challenge in deploying learning-based systems is scale: acquiring enough data for the robot to effectively generalize broadly.

Imitation learning, in particular, has remained a stable and powerful approach for robot learning, but critically relies on expert operators for data collection.
In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation.
To accomplish this, we cast the problem of imitation with autonomous improvement into a multi-task setting.

Built on

D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in Neural Information Processing Systems (NIPS) , 1989, pp. 305–313
1989
Earlier work this paper cites.
L. P. Kaelbling, “Learning to achieve goals,” in Proceedings of the 13th International Joint Conference on Artificial Intelligence 1993 , 1993
1993
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press Cambridge, 1998, vol. 1, no. 1
1998
Earlier work this paper cites.
A. Y. Ng and S. J. Russell, “Algorithms for inverse reinforcement learning.” in International Conference on Machine Leanring (ICML) , 2000
2000
Earlier work this paper cites.
S. Schaal, A. Ijspeert, and A. Billard, “Computational approaches to motor learning by imitation,” Philosophical Transactions of the Royal Society of London B: Biological Sciences , 2003
2003
Earlier work this paper cites.
P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in International Conference on Machine Learning (ICML) . ACM, 2004, p. 1
2004
Earlier work this paper cites.
N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich, “Maximum margin planning,” in International Conference on Machine Learning (ICML) . ACM, 2006
2006
Earlier work this paper cites.
R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA , 2006
2006
Earlier work this paper cites.
S. Ross and D. Bagnell, “Efficient reductions for imitation learning,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics , 2010, pp. 661–668
2010
Earlier work this paper cites.
J. Kober and J. Peters, “Reinforcement learning in robotics: A survey,” in Reinforcement Learning . Springer, 2012, pp. 579–610
2012
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602 , 2013
Original
2013
Earlier work this paper cites.
M. P. Deisenroth, P. Englert, J. Peters, and D. Fox, “Multi-task policy search for robotics,” in International Conference on Robotics and Automation (ICRA) , 2014
2014
Earlier work this paper cites.
M. Wulfmeier, P. Ondruska, and I. Posner, “Maximum entropy deep inverse reinforcement learning,” in Neural Information Processing Systems Conference, Deep Reinforcement Learning Workshop , vol. abs/1507.04888, 2015
Original
2015
Earlier work this paper cites.

Similar

D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations (ICLR) , 2015
2015
Cited alongside, same era.
C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in International Conference on Machine Learning (ICML) , 2016
2016
Cited alongside, same era.
J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Neural Information Processing Systems (NIPS) , 2016
2016
Cited alongside, same era.
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl 2 : Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779 , 2016
Original
2016
Cited alongside, same era.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” GitHub repository , 2016
2016
Cited alongside, same era.
T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” arXiv preprint arXiv:1710.04615 , 2017
Original
2017
Cited alongside, same era.
W. Sun, A. Venkatraman, G. J. Gordon, B. Boots, and J. A. Bagnell, “Deeply aggrevated: Differentiable imitation learning for sequential prediction,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 3309–3318
2017
Cited alongside, same era.
Y. Duan, M. Andrychowicz, B. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” Neural Information Processing Systems (NIPS) , 2017
2017
Cited alongside, same era.
C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” Conference on Robot Learning (CoRL) , 2017
2017
Cited alongside, same era.
M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” in Neural Information Processing Systems (NIPS) , 2017, pp. 5055–5065
2017
Cited alongside, same era.
P. Rauber, A. Ummadisingu, F. Mutz, and J. Schmidhuber, “Hindsight policy gradients,” arXiv preprint arXiv:1711.06006 , 2017
Original
2017
Cited alongside, same era.
Y. Duan, M. Andrychowicz, B. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” Neural Information Processing Systems (NIPS) , 2017
2017
Cited alongside, same era.

Then

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, V. K. Jie Tan, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” Tech. Rep., 2018
2018
Later among the works it cites.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in Thirty-Second AAAI Conference on Artificial Intelligence , 2018
2018
Later among the works it cites.
A. Irpan, “Deep reinforcement learning doesn’t work yet,” https://www.alexirpan.com/2018/02/14/rl-hard.html , 2018
2018
Later among the works it cites.
T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” Robotics: Science and Systems (RSS) , 2018
2018
Later among the works it cites.
S. James, M. Bloesch, and A. J. Davison, “Task-embedded control networks for few-shot imitation learning,” arXiv preprint arXiv:1810.03237 , 2018
Original
2018
Later among the works it cites.
D. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” 2018
2018
Later among the works it cites.
T. L. Paine, S. G. Colmenarejo, Z. Wang, S. Reed, Y. Aytar, T. Pfaff, M. W. Hoffman, G. Barth-Maron, S. Cabi, D. Budden et al. , “One-shot high-fidelity imitation: Training large-scale deep nets with rl,” arXiv preprint arXiv:1810.05017 , 2018
Original
2018
Later among the works it cites.
K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. A. Riedmiller, “Learning an embedding space for transferable robot skills,” in International Conference on Learning Representations, (ICLR) , 2018
2018
Later among the works it cites.
X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee, “Learning 6-dof grasping interaction via deep geometry-aware 3d representations,” in International Conference on Robotics and Automation (ICRA) , 2018
2018
Later among the works it cites.
A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, “End-to-end robotic reinforcement learning without reward engineering,” Robotics: Science and Systems , 2019
2019
Later among the works it cites.
G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of real-world reinforcement learning,” arXiv preprint arXiv:1904.12901 , 2019
Original
2019
Later among the works it cites.
T. Yu, G. Shevchuk, D. Sadigh, and C. Finn, “Unsupervised visuomotor control through distributional planning networks,” arXiv preprint arXiv:1902.05542 , 2019
Original
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…