2019

Watch, Try, Learn: Meta-Learning from Demonstrations and Reward

Zhou, Allan, Jang, Eric, Kappler, Daniel et al.

Understand

Imitation learning allows agents to learn complex behaviors from demonstrations.

However, learning a complex vision-based task may require an impractical number of demonstrations.
Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks.
In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy.

Built on

Evolutionary principles in self-referential learning
Jürgen Schmidhuber · 1987
Earlier work this paper cites.
On the optimization of a synaptic learning rule
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei · 1992
Earlier work this paper cites.
Mixture density networks
Christopher M Bishop · 1994
Earlier work this paper cites.
Learning to learn
Sebastian Thrun and Lorien Pratt · 1998
Earlier work this paper cites.
Learning to learn using gradient descent
Sepp Hochreiter, A Younger, and Peter Conwell · 2001
Earlier work this paper cites.
Policy gradient methods for robotics
Jan Peters and Stefan Schaal · 2006
Earlier work this paper cites.
Policy search for motor primitives in robotics
Jens Kober and Jan R Peters · 2009
Earlier work this paper cites.
Robot motor skill coordination with em-based reinforcement learning
Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell · 2010
Earlier work this paper cites.
Integrating reinforcement learning with human demonstrations of varying ability
Matthew E Taylor, Halit Bener Suay, and Sonia Chernova · 2011
Earlier work this paper cites.
Reinforcement learning in robotics: A survey
Jens Kober, J Andrew Bagnell, and Jan Peters · 2013
Earlier work this paper cites.
Reinforcement learning from demonstration through shaping
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Openai gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Earlier work this paper cites.
Pybullet, a python module for physics simulation for games, robotics and machine learning
E Coumans and Y Bai · 2016
Earlier work this paper cites.

Similar

Rl 2 : Fast reinforcement learning via slow reinforcement learning
Original
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel · 2016
Cited alongside, same era.
Continuous deep q-learning with model-based acceleration
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine · 2016
Cited alongside, same era.
End-to-end training of deep visuomotor policies
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
Exploration from demonstration for interactive reinforcement learning
Kaushik Subramanian, Charles L Isbell Jr, and Andrea L Thomaz · 2016
Cited alongside, same era.
Learning to reinforcement learn
Original
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick · 2016
Cited alongside, same era.
One-shot imitation from observing humans via domain-adaptive meta-learning
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine · 2016
Cited alongside, same era.
One-shot imitation learning
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.
Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim · 2017
Cited alongside, same era.
Learning to learn: Meta-critic networks for sample efficient learning
Original
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang · 2017
Cited alongside, same era.
Learning to Learn with Gradients
Chelsea Finn · 2018
Cited alongside, same era.
TF-Agents: A library for reinforcement learning in tensorflow, 2018
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo · 2018
Cited alongside, same era.

Then

Deep q-learning from demonstrations
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, et al · 2018
Later among the works it cites.
Evolved policy gradients
Original
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel · 2018
Later among the works it cites.
Task-embedded control networks for few-shot imitation learning
Original
Stephen James, Michael Bloesch, and Andrew J Davison · 2018
Later among the works it cites.
Hierarchical imitation and reinforcement learning
Original
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III · 2018
Later among the works it cites.
A simple neural attentive meta-learner
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel · 2018
Later among the works it cites.
Overcoming exploration in reinforcement learning with demonstrations
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel · 2018
Later among the works it cites.
One-shot high-fidelity imitation: Training large-scale deep nets with rl
Original
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed, Yusuf Aytar, Tobias Pfaff, Matt W Hoffman, Gabriel Barth-Maron, Serkan Cabi, David Budden, et al · 2018
Later among the works it cites.
Learning complex dexterous manipulation with deep reinforcement learning and demonstrations
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine · 2018
Later among the works it cites.
A tutorial on thompson sampling
Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al · 2018
Later among the works it cites.
Meta reinforcement learning with latent variable gaussian processes
Original
Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth · 2018
Later among the works it cites.
Some considerations on learning to explore via meta-reinforcement learning
Original
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever · 2018
Later among the works it cites.
Truncated horizon policy search: Combining reinforcement learning & imitation learning
Wen Sun, J Andrew Bagnell, and Byron Boots · 2018
Later among the works it cites.
Learning to adapt in dynamic, real-world environments through meta-reinforcement learning
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2019
Closest in time.
Efficient off-policy meta-reinforcement learning via probabilistic context variables
Original
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…