2018

One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

Paine, Tom Le, Colmenarejo, Sergio Gómez, Wang, Ziyu et al.

Understand

Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt.

Humans use this ability to quickly solve a task instance, and to bootstrap learning of new tasks.
Achieving these abilities in autonomous agents is an open problem.
In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap.

Built on

Alvinn: An autonomous land vehicle in a neural network
Dean A Pomerleau · 1989
Earlier work this paper cites.
Robot learning from demonstration
Christopher G Atkeson and Stefan Schaal · 1997
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
Andrew Y Ng, Stuart J Russell, et al · 2000
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng · 2004
Earlier work this paper cites.
Causal knowledge and imitation/emulation switching in chimpanzees (pan troglodytes) and children (homo sapiens)
Victoria Horner and Andrew Whiten · 2005
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey · 2008
Earlier work this paper cites.
From over-imitation to super-copying: Adults imitate causally irrelevant aspects of tool use with higher fidelity than young children
Nicola McGuigan, Jenny Makinson, and Andrew Whiten · 2011
Earlier work this paper cites.
Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup · 2011
Earlier work this paper cites.
Playing atari with deep reinforcement learning
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Earlier work this paper cites.
Fast and accurate deep network learning by exponential linear units (elus)
Original
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Imitation and innovation: The dual engines of cultural learning
Cristine H Legare and Mark Nielsen · 2015
Earlier work this paper cites.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Guided cost learning: Deep inverse optimal control via policy optimization
Chelsea Finn, Sergey Levine, and Pieter Abbeel · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Earlier work this paper cites.

Similar

Playing atari games with deep reinforcement learning and human checkpoint replay
Original
Ionel-Alexandru Hosu and Traian Rebedea · 2016
Cited alongside, same era.
Unsupervised perceptual rewards for imitation learning
Original
Pierre Sermanet, Kelvin Xu, and Sergey Levine · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
Instance normalization: the missing ingredient for fast stylization. corr abs/1607.08022 (2016)
Original
D Ulyanov, A Vedaldi, and VS Lempitsky · 2016
Cited alongside, same era.
Hindsight experience replay
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.
A distributional perspective on reinforcement learning
Original
Marc G Bellemare, Will Dabney, and Rémi Munos · 2017
Cited alongside, same era.
The intentional unintentional agent: Learning to solve many continuous control tasks simultaneously
Original
Serkan Cabi, Sergio Gómez Colmenarejo, Matthew W Hoffman, Misha Denil, Ziyu Wang, and Nando De Freitas · 2017
Cited alongside, same era.
One-shot imitation learning
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.
One-shot visual imitation learning via meta-learning
Original
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
Deep q-learning from demonstrations
Original
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, et al · 2017
Cited alongside, same era.
Batch renormalization: Towards reducing minibatch dependence in batch-normalized models
Sergey Ioffe · 2017
Cited alongside, same era.
Imitation from observation: Learning to imitate behaviors from raw video via context translation
Original
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.

Then

Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards
Original
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin A Riedmiller · 2017
Later among the works it cites.
Robust imitation of diverse behaviors
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess · 2017
Later among the works it cites.
Playing hard exploration games by watching youtube
Original
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas · 2018
Closest in time.
Distributed distributional deterministic policy gradients
Original
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap · 2018
Closest in time.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures
Original
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al · 2018
Closest in time.
Synthesizing programs for images using reinforced adversarial learning
Original
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Eslami, and Oriol Vinyals · 2018
Closest in time.
Distributed prioritized experience replay
Original
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver · 2018
Closest in time.
Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation
Original
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al · 2018
Closest in time.
Visual reinforcement learning with imagined goals
Original
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine · 2018
Closest in time.
Learning Dexterous In-Hand Manipulation
OpenAI, :, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba · 2018
Closest in time.
Zero-shot visual imitation
Original
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell · 2018
Closest in time.
Deepmimic: Example-guided deep reinforcement learning of physics-based character skills
Original
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne · 2018
Closest in time.
Backplay:” man muss immer umkehren”
Original
Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alex Peysakhovich, Kyunghyun Cho, and Joan Bruna · 2018
Closest in time.
Learning by playing-solving sparse reward tasks from scratch
Original
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg · 2018
Closest in time.
Time-contrastive networks: Self-supervised learning from video
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine · 2018
Closest in time.
Behavioral cloning from observation
Original
Faraz Torabi, Garrett Warnell, and Peter Stone · 2018
Closest in time.
Reinforcement and imitation learning for diverse visuomotor skills
Original
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…