2019

Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

Yang, Chao, Ma, Xiaojian, Huang, Wenbing et al.

Understand

This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations.

In contrast to Learning from Demonstration (LfD) that involves both action and state supervision, LfO is more practical in leveraging previously inapplicable resources (e.g.
videos), yet more challenging due to the incomplete expert guidance.
In this paper, we investigate LfO and its difference with LfD in both theoretical and practical perspectives.

Built on

On adaptive inverse dynamics control of rigid robots
Mark W Spong and Romeo Ortega · 1990
Earlier work this paper cites.
Markov decision processes: discrete stochastic dynamic programming
Martin L Puterman · 1994
Earlier work this paper cites.
Robot learning from demonstration
Christopher G Atkeson and Stefan Schaal · 1997
Earlier work this paper cites.
Learning from demonstration
Stefan Schaal · 1997
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 1998
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
Andrew Y Ng, Stuart J Russell, et al · 2000
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng · 2004
Earlier work this paper cites.
Learning tasks from observation and practice
Darrin C Bentivegna, Christopher G Atkeson, and Gordon Cheng · 2004
Earlier work this paper cites.
Springer handbook of robotics
Bruno Siciliano and Oussama Khatib · 2008
Earlier work this paper cites.
Apprenticeship learning using linear programming
Umar Syed, Michael Bowling, and Robert E Schapire · 2008
Earlier work this paper cites.
Efficient reductions for imitation learning
Stéphane Ross and Drew Bagnell · 2010
Earlier work this paper cites.
Model learning for robot control: a survey
Duy Nguyen-Tuong and Jan Peters · 2011
Earlier work this paper cites.
Agnostic system identification for model-based reinforcement learning
Stephane Ross and J Andrew Bagnell · 2012
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.

Similar

Maximum mean discrepancy imitation learning
Beomjoon Kim and Joelle Pineau · 2013
Cited alongside, same era.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Cited alongside, same era.
End to end learning for self-driving cars
Original
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al · 2016
Cited alongside, same era.
Openai gym, 2016
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
Benchmarking deep reinforcement learning for continuous control
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel · 2016
Cited alongside, same era.
A machine learning approach to visual perception of forest trails for mobile robots
Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Cited alongside, same era.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Cited alongside, same era.
f-gan: Training generative neural samplers using variational divergence minimization
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka · 2016
Cited alongside, same era.
Pybullet, a python module for physics simulation in robotics, games and machine learning
Erwin Coumans and Yunfei Bai · 2017
Cited alongside, same era.
One-shot imitation learning
Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba · 2017
Cited alongside, same era.

Then

Learning dexterous in-hand manipulation
Original
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al · 2018
Later among the works it cites.
Mutual information neural estimation
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm · 2018
Later among the works it cites.
Learning robust rewards with adversarial inverse reinforcement learning
Justin Fu, Katie Luo, and Sergey Levine · 2018
Later among the works it cites.
Policy optimization with demonstrations
Bingyi Kang, Zequn Jie, and Jiashi Feng · 2018
Later among the works it cites.
Imitation learning via kernel mean embedding
Kee-Eung Kim and Hyun Soo Park · 2018
Later among the works it cites.
Imitation from observation: Learning to imitate behaviors from raw video via context translation
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine · 2018
Later among the works it cites.
Deepmimic: Example-guided deep reinforcement learning of physics-based character skills
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne · 2018
Later among the works it cites.
Learning to walk via deep reinforcement learning
Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, and Sergey Levine · 2019
Closest in time.
Learning deep representations by mutual information estimation and maximization
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio · 2019
Closest in time.
Task transfer by preference-based cost learning
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu · 2019
Closest in time.
Maximum entropy generators for energy-based models
Original
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio · 2019
Closest in time.
Variational discriminator bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine · 2019
Closest in time.
Provably efficient imitation learning from observation alone
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell · 2019
Closest in time.
Imitation learning from video by leveraging proprioception
Faraz Torabi, Garrett Warnell, and Peter Stone · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…