2017

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Vecerik, Mel, Hester, Todd, Scholz, Jonathan et al.

Understand

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards.

We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations.
Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism.
Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics.

ALVINN: An autonomous land vehicle in a neural network
D. A. Pomerleau · 1989
Earlier work this paper cites.
Introduction to reinforcement learning
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
A. Y. Ng, D. Harada, and S. Russell · 1999
Earlier work this paper cites.
A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems
E. Todorov and W. Li · 2005
Earlier work this paper cites.
Imitation learning for locomotion and manipulation
N. Ratliff, J. A. Bagnell, and S. S. Srinivasa · 2007
Earlier work this paper cites.
A game-theoretic approach to apprenticeship learning
U. Syed and R. E. Schapire · 2007
Earlier work this paper cites.
Apprenticeship learning using linear programming
U. Syed, M. Bowling, and R. E. Schapire · 2008
Earlier work this paper cites.

Maximum entropy inverse reinforcement learning
Original
B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey · 2008
Cited alongside, same era.
A reduction of imitation learning and structured prediction to no-regret online learning
S. Ross, G. J. Gordon, and J. A. Bagnell · 2011
Cited alongside, same era.
Mujoco: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Cited alongside, same era.
A cascaded supervised learning approach to inverse reinforcement learning
E. Klein, B. Piot, M. Geist, and O. Pietquin · 2013
Cited alongside, same era.
Guided policy search
S. Levine and V. Koltun · 2013
Cited alongside, same era.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis · 2015
Cited alongside, same era.
Trust region policy optimization
J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel · 2015
Cited alongside, same era.

Guided cost learning: Deep inverse optimal control via policy optimization
C. Finn, S. Levine, and P. Abbeel · 2016
Later among the works it cites.
Continuous deep q-learning with model-based acceleration
S. Gu, T. Lillicrap, I. Sutskever, and S. Levine · 2016
Later among the works it cites.
Generative adversarial imitation learning
J. Ho and S. Ermon · 2016
Later among the works it cites.
Continuous control with deep reinforcement learning
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra · 2016
Later among the works it cites.
Prioritized experience replay
Original
T. Schaul, J. Quan, I. Antonoglou, and D. Silver · 2016
Later among the works it cites.
Learning from demonstrations for real world reinforcement learning
Original
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, et al · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…