2019

Imitation Learning as $f$-Divergence Minimization

Ke, Liyiming, Choudhury, Sanjiban, Barnes, Matt et al.

Understand

We address the problem of imitation learning with multi-modal demonstrations.

Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them.
We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes.
Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection.

Built on

Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng · 2004
Earlier work this paper cites.
Information theory and statistics: A tutorial
Imre Csiszár and Paul C Shields · 2004
Earlier work this paper cites.
Maximum margin planning
Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich · 2006
Earlier work this paper cites.
On divergences and informations in statistics and information theory
Friedrich Liese and Igor Vajda · 2006
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey · 2008
Earlier work this paper cites.
A game-theoretic approach to apprenticeship learning
Umar Syed and Robert E Schapire · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning · 2009
Earlier work this paper cites.
Learning to search: Functional gradient techniques for imitation learning
Nathan D Ratliff, David Silver, and J Andrew Bagnell · 2009
Earlier work this paper cites.
Estimating divergence functionals and the likelihood ratio by convex risk minimization
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan · 2010
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell · 2011
Earlier work this paper cites.
Relative entropy inverse reinforcement learning
Abdeslam Boularias, Jens Kober, and Jan Peters · 2011
Earlier work this paper cites.
Apprenticeship learning about multiple intentions
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman · 2011
Earlier work this paper cites.
Bayesian multitask inverse reinforcement learning
Christos Dimitrakakis and Constantin A Rothkopf · 2011
Earlier work this paper cites.
Statistical analysis of kernel-based least-squares density-ratio estimation
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama · 2012
Earlier work this paper cites.
Learning monocular reactive uav control in cluttered natural environments
Stéphane Ross, Narek Melik-Barkhudarov, Kumar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J Andrew Bagnell, and Martial Hebert · 2013
Earlier work this paper cites.

Similar

Learning from limited demonstrations
Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup · 2013
Cited alongside, same era.
Reinforcement and imitation learning via interactive no-regret learning
Original
Stephane Ross and J Andrew Bagnell · 2014
Cited alongside, same era.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Cited alongside, same era.
An invitation to imitation
J. Andrew (Drew) Bagnell · 2015
Cited alongside, same era.
Maximum entropy deep inverse reinforcement learning
Original
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner · 2015
Cited alongside, same era.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Cited alongside, same era.
f-gan: Training generative neural samplers using variational divergence minimization
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka · 2016
Cited alongside, same era.
Learning from humans
Aude G Billard, Sylvain Calinon, and Rüdiger Dillmann · 2016
Cited alongside, same era.
Shiv: Reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces
Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D Dragan, and Ken Goldberg · 2016
Cited alongside, same era.
Model-free imitation learning with policy optimization
Jonathan Ho, Jayesh Gupta, and Stefano Ermon · 2016
Cited alongside, same era.
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel · 2016
Cited alongside, same era.
Infogail: Interpretable imitation learning from visual demonstrations
Yunzhu Li, Jiaming Song, and Stefano Ermon · 2017
Cited alongside, same era.

Then

Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations
Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin Jamieson, Anca Dragan, and Ken Goldberg · 2017
Later among the works it cites.
Bridging the gap between imitation learning and inverse reinforcement learning
Bilal Piot, Matthieu Geist, and Olivier Pietquin · 2017
Later among the works it cites.
Learning robust rewards with adversarial inverse reinforcement learning
Original
Justin Fu, Katie Luo, and Sergey Levine · 2017
Later among the works it cites.
Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim · 2017
Later among the works it cites.
f-divergence constrained policy improvement
Original
Boris Belousov and Jan Peters · 2017
Later among the works it cites.
An algorithmic perspective on imitation learning
Original
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters · 2018
Later among the works it cites.
Truncated horizon policy search: Combining reinforcement learning & imitation learning
Original
Wen Sun, J Andrew Bagnell, and Byron Boots · 2018
Later among the works it cites.
Fast policy learning through imitation and reinforcement
Original
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots · 2018
Later among the works it cites.
Sample-efficient imitation learning via generative adversarial nets
Original
Lionel Blondé and Alexandros Kalousis · 2018
Later among the works it cites.
Adversarial imitation via variational inverse reinforcement learning
Original
Ahmed H Qureshi and Michael C Yip · 2018
Later among the works it cites.
Sfv: Reinforcement learning of physical skills from videos
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine · 2018
Later among the works it cites.
R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting
Nicholas Rhinehart, Kris M. Kitani, and Paul Vernaza · 2018
Later among the works it cites.
Understanding the relation between maximum-entropy inverse reinforcement learning and behaviour cloning
Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard Zemel · 2018
Later among the works it cites.
Variational f-divergence minimization
Original
Mingtian Zhang, Thomas Bird, Raza Habib, Tianlin Xu, and David Barber · 2019
Closest in time.
Provably efficient imitation learning from observation alone
Original
Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…