2018

One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks

Yu, Tianhe, Abbeel, Pieter, Levine, Sergey et al.

Understand

We consider the problem of learning multi-stage vision-based tasks on a real robot from a single video of a human performing the task, while leveraging demonstration data of subtasks with other objects.

This problem presents a number of major challenges.
Video demonstrations without teleoperation are easy for humans to provide, but do not provide any direct supervision.
Learning policies from raw pixels enables full generality but calls for large function approximators with many parameters to be learned.

Built on

S. Calinon, P. Evrard, E. Gribovskaya, A. Billard, and A. Kheddar, “Learning collaborative manipulation tasks by demonstration using a haptic interface,” in International Conference on Advanced Robotics (ICAR) , 2009
2009
Earlier work this paper cites.
J. Butterfield, S. Osentoski, G. Jay, and O. C. Jenkins, “Learning from demonstration using a multi-valued function regressor for time-series data,” in Humanoid Robots (Humanoids), 2010 10th IEEE-RAS International Conference on . IEEE, 2010, pp. 328–333
2010
Earlier work this paper cites.
D. H. Grollman and O. C. Jenkins, “Incremental learning of subtasks from unsegmented demonstration,” in Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on . IEEE, 2010, pp. 261–266
2010
Earlier work this paper cites.
P. Pastor, L. Righetti, M. Kalakrishnan, and S. Schaal, “Online movement adaptation based on previous sensor experiences,” in International Conference on Intelligent Robots and Systems (IROS) , 2011
2011
Earlier work this paper cites.
F. Meier, E. Theodorou, F. Stulp, and S. Schaal, “Movement segmentation using a primitive library,” in Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on . IEEE, 2011, pp. 3407–3412
2011
Earlier work this paper cites.
S. Ross, G. J. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning.” in AISTATS , 2011
2011
Earlier work this paper cites.
B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz, “Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective,” in International Conference on Human-Robot Interaction , 2012
2012
Earlier work this paper cites.
J. Kober, A. Wilhelm, E. Oztop, and J. Peters, “Reinforcement learning to adjust parametrized motor primitives to new situations,” Autonomous Robots , 2012
2012
Earlier work this paper cites.
G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto, “Robot learning from demonstration by constructing skill trees,” The International Journal of Robotics Research , pp. 360–375, 2012
2012
Earlier work this paper cites.
P. Pastor, M. Kalakrishnan, L. Righetti, and S. Schaal, “Towards associative skill memories,” in Humanoid Robots (Humanoids), 2012 12th IEEE-RAS International Conference on . IEEE, 2012, pp. 309–315
2012
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in International Conference on Intelligent Robots and Systems (IROS) , 2012
2012
Earlier work this paper cites.
S. Niekum, S. Chitta, A. G. Barto, B. Marthi, and S. Osentoski, “Incremental semantically grounded learning from demonstration.” in Robotics: Science and Systems , vol. 9. Berlin, Germany, 2013
2013
Earlier work this paper cites.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research , vol. 15, no. 1, pp. 1929–1958, 2014
2014
Earlier work this paper cites.

Similar

S. Manschitz, J. Kober, M. Gienger, and J. Peters, “Learning movement primitive attractor goals and sequential skills from kinesthetic demonstrations,” Robotics and Autonomous Systems , vol. 74, pp. 97–107, 2015
2015
Cited alongside, same era.
Y. Yang, Y. Li, C. Fermüller, and Y. Aloimonos, “Robot learning manipulation action plans by” watching” unconstrained videos from the world wide web.” in AAAI Conference on Artificial Intelligence (AAAI) , 2015
2015
Cited alongside, same era.
S. Manschitz, J. Kober, M. Gienger, and J. Peters, “Probabilistic progress prediction and sequencing of concurrent movement primitives,” in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on . IEEE, 2015, pp. 449–455
2015
Cited alongside, same era.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations (ICLR) , 2015
2015
Cited alongside, same era.
A. Murali, A. Garg, S. Krishnan, F. T. Pokorny, P. Abbeel, T. Darrell, and K. Goldberg, “Tsc-dl: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with deep learning,” in Robotics and Automation (ICRA), 2016 IEEE International Conference on . IEEE, 2016
2016
Cited alongside, same era.
J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv:1607.06450 , 2016
Original
2016
Cited alongside, same era.
R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine, “Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration,” arXiv:1707.02920 , 2017
Original
2017
Cited alongside, same era.
P. Sermanet, C. Lynch, J. Hsu, and S. Levine, “Time-contrastive networks: Self-supervised learning from multi-view observation,” arXiv:1704.06888 , 2017
Original
2017
Cited alongside, same era.
B. Stadie, P. Abbeel, and I. Sutskever, “Third-person imitation learning,” International Conference on Learning Representations (ICLR) , 2017
2017
Cited alongside, same era.
D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese, “Neural task programming: Learning to generalize across hierarchical tasks,” arXiv preprint arXiv:1710.01813 , 2017
Original
2017
Cited alongside, same era.
D. Wilbers, R. Lioutikov, and J. Peters, “Context-driven movement primitive adaptation,” in International Conference on Robotics and Automation (ICRA) , 2017
2017
Cited alongside, same era.
A. Nguyen, D. Kanoulas, L. Muratore, D. G. Caldwell, and N. G. Tsagarakis, “Translating videos to commands for robotic manipulation with deep recurrent neural networks,” arXiv:1710.00290 , 2017
Original
2017
Cited alongside, same era.

Then

E. E. Aksoy, E. Ovchinnikova, A. Orhan, Y. Yang, and T. Asfour, “Unsupervised linking of visual features to textual descriptions in long manipulation activities,” Robotics and Automation Letters , 2017
2017
Later among the works it cites.
Y. Duan, M. Andrychowicz, B. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,” Neural Information Processing Systems (NIPS) , 2017
2017
Later among the works it cites.
K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim, “Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets,” in Advances in Neural Information Processing Systems , 2017, pp. 1235–1245
2017
Later among the works it cites.
C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” International Conference on Machine Learning (ICML) , 2017
2017
Later among the works it cites.
P. Ramachandran, B. Zoph, and Q. V. Le, “Swish: a self-gated activation function,” arXiv preprint arXiv:1710.05941 , 2017
Original
2017
Later among the works it cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Later among the works it cites.
T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” International Conference on Robotics and Automation (ICRA) , 2018
2018
Closest in time.
R. Rahmatizadeh, P. Abolghasemi, A. Behal, and L. Bölöni, “Learning real manipulation tasks from virtual demonstrations using lstm,” AAAI Conference on Artificial Intelligence (AAAI) , 2018
2018
Closest in time.
Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” International Conference on Robotics and Automation (ICRA) , 2018
2018
Closest in time.
T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,” Robotics: Science and Systems (R:SS) , 2018
2018
Closest in time.
K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner, “Taco: Learning task decomposition via temporal alignment for control,” arXiv preprint arXiv:1803.01840 , 2018
Original
2018
Closest in time.
D.-A. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles, “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” arXiv preprint arXiv:1807.03480 , 2018
Original
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…