2020

The MAGICAL Benchmark for Robust Imitation

Toyer, Sam, Shah, Rohin, Critch, Andrew et al.

Understand

Imitation Learning (IL) algorithms are typically evaluated in the same environment that was used to create demonstrations.

This rewards precise reproduction of demonstrations in one particular environment, but provides little information about how robustly an algorithm can generalise the demonstrator's intent to substantially different deployment settings.
This paper presents the MAGICAL benchmark suite, which permits systematic evaluation of generalisation by quantifying robustness to different kinds of distribution shift that an IL algorithm is likely to encounter in practice.
Using the MAGICAL suite, we confirm that existing IL algorithms overfit significantly to the context in which demonstrations are provided.

Built on

A RANSAC-based approach to model fitting and its application to finding cylinders in range data
Robert C Bolles and Martin A Fischler · 1981
Earlier work this paper cites.
Acquiring recursive and iterative concepts with explanation-based learning
Iude W Shavlik · 1990
Earlier work this paper cites.
Deferred imitation across changes in context and object: Memory and generalization in 14-month-old infants
Sandra B Barnat, Pamela J Klein, and Andrew N Meltzoff · 1996
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng · 2004
Earlier work this paper cites.
A decision-theoretic model of assistance
Alan Fern, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli · 2007
Earlier work this paper cites.
Survey: Robot programming by demonstration
Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal · 2008
Earlier work this paper cites.
Apprenticeship learning about multiple intentions
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman · 2011
Earlier work this paper cites.
Bayesian multitask inverse reinforcement learning
Christos Dimitrakakis and Constantin A Rothkopf · 2011
Earlier work this paper cites.
Nonparametric bayesian inverse reinforcement learning for multiple reward functions
Jaedeug Choi and Kee-Eung Kim · 2012
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
OpenAI Gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Earlier work this paper cites.
Cooperative inverse reinforcement learning
Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan · 2016
Earlier work this paper cites.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Earlier work this paper cites.
Model-free imitation learning with policy optimization
Jonathan Ho, Jayesh Gupta, and Stefano Ermon · 2016
Earlier work this paper cites.

Similar

One-shot imitation learning
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.
Learning robust rewards with adversarial inverse reinforcement learning
Original
Justin Fu, Katie Luo, and Sergey Levine · 2017
Cited alongside, same era.
Improved training of Wasserstein GANs
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville · 2017
Cited alongside, same era.
Relational inductive biases, deep learning, and graph networks
Original
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al · 2018
Cited alongside, same era.
Quantifying generalization in reinforcement learning
Original
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman · 2018
Cited alongside, same era.
Multi-task maximum entropy inverse reinforcement learning
Original
Adam Gleave and Oliver Habryka · 2018
Cited alongside, same era.
Task-embedded control networks for few-shot imitation learning
Stephen James, Michael Bloesch, and Andrew J Davison · 2018
Cited alongside, same era.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Brenden M Lake and Marco Baroni · 2018
Cited alongside, same era.
Gotta learn fast: A new benchmark for generalization in RL
Original
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman · 2018
Cited alongside, same era.
Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow
Original
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine · 2018
Cited alongside, same era.
Behavioral cloning from observation
Faraz Torabi, Garrett Warnell, and Peter Stone · 2018
Cited alongside, same era.
Solving Rubik’s Cube with a robot hand
Original
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al · 2019
Cited alongside, same era.

Then

Rlbench: The robot learning benchmark & learning environment
Original
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison · 2019
Later among the works it cites.
Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson · 2019
Later among the works it cites.
Simitate: A hybrid imitation learning benchmark
Original
Raphael Memmesheimer, Ivanna Mykhalchyshyna, Viktor Seib, and Dietrich Paulus · 2019
Later among the works it cites.
Adversarial imitation via variational inverse reinforcement learning
Ahmed H Qureshi, Byron Boots, and Michael C Yip · 2019
Later among the works it cites.
rlpyt: A research code base for deep reinforcement learning in pytorch
Original
Adam Stooke and Pieter Abbeel · 2019
Later among the works it cites.
Learning a prior over intent via meta-inverse reinforcement learning
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn · 2019
Later among the works it cites.
Meta-inverse reinforcement learning with probabilistic context variables
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon · 2019
Later among the works it cites.
Task-relevant adversarial imitation learning
Original
Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarej, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang · 2019
Later among the works it cites.
Safe imitation learning via fast bayesian reward inference from preferences
Original
Daniel S Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum · 2020
Closest in time.
Environmental drivers of systematicity and generalization in a situated agent
Felix Hill, Andrew K Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro · 2020
Closest in time.
Thieves on Sesame Street! Model extraction of BERT-based APIs
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit Iyyer · 2020
Closest in time.
A benchmark for systematic generalization in grounded language understanding
Original
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake · 2020
Closest in time.
Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…