2017

Reverse Curriculum Generation for Reinforcement Learning

Florensa, Carlos, Held, David, Wulfmeier, Markus et al.

Understand

Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration.

For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock.
These goal-oriented tasks present a considerable challenge for reinforcement learning, since their natural reward function is sparse and prohibitive amounts of exploration are required to reach the goal and receive some learning signal.
Past approaches tackle these problems by exploiting expert demonstrations or by manually designing a task-specific reward shaping function to guide the learning agent.

Built on

Complexity and Cooperation in Q-Learning
S. D. Whitehead · 1991
Earlier work this paper cites.
Dynamic programming and optimal control , volume 1
D. P. Bertsekas, D. P. Bertsekas, D. P. Bertsekas, and D. P. Bertsekas · 1995
Earlier work this paper cites.
Purposive behavior acquisition for a real robot by Vision-Based reinforcement learning
M. Asada, S. Noda, S. Tawaratsumida, and K. Hosoda · 1996
Earlier work this paper cites.
Reinforcement Learning: An Introduction
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
A. Y. Ng, D. Harada, and S. Russell · 1999
Earlier work this paper cites.
Sequential composition of dynamically dexterous robot behaviors
R. R. Burridge, A. A. Rizzi, and D. E. Koditschek · 1999
Earlier work this paper cites.
RRT-connect: An efficient approach to single-query path planning
J. Kuffner and S. LaValle · 2000
Earlier work this paper cites.
Approximately Optimal Approximate Reinforcement Learning
S. Kakade and J. Langford · 2002
Earlier work this paper cites.
A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes
M. Kearns, Y. Mansour, and A. Y. Ng · 2002
Earlier work this paper cites.
Policy search by dynamic programming
J. A. Bagnell, S. Kakade, A. Y. Ng, and J. Schneider · 2003
Earlier work this paper cites.
Learning tetris using the noisy cross-entropy method
I. Szita and A. Lörincz · 2006
Earlier work this paper cites.
Reinforcement learning of motor skills with policy gradients
J. Peters and S. Schaal · 2008
Earlier work this paper cites.
Curriculum learning
Y. Bengio, J. Louradour, R. Collobert, and J. Weston · 2009
Earlier work this paper cites.
Self-paced learning for latent variable models
M. P. Kumar, B. Packer, and D. Koller · 2010
Earlier work this paper cites.

Similar

Lqr-trees: Feedback motion planning via sums-of-squares verification
R. Tedrake, I. R. Manchester, M. Tobenkin, and J. W. Roberts · 2010
Cited alongside, same era.
Curriculum learning for motor skills
A. Karpathy and M. Van De Panne · 2012
Cited alongside, same era.
Continually adding self-invented problems to the repertoire: First experiments with POWERPLAY
R. K. Srivastava, B. R. Steunebrink, M. Stollenga, and J. Schmidhuber · 2012
Cited alongside, same era.
Mujoco: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Cited alongside, same era.
POWER P LAY : Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem
J. Schmidhuber · 2013
Cited alongside, same era.
Active learning of inverse models with intrinsically motivated goal exploration in robots
A. Baranes and P.-Y. Oudeyer · 2013
Cited alongside, same era.
A survey on policy search for robotics
M. P. Deisenroth, G. Neumann, J. Peters, et al · 2013
Cited alongside, same era.
Learning to execute
Original
W. Zaremba and I. Sutskever · 2014
Cited alongside, same era.
High-Dimensional continuous control using generalized advantage estimation
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al · 2015
Cited alongside, same era.
Continuous control with deep reinforcement learning
Original
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra · 2015
Cited alongside, same era.
Scheduled sampling for sequence prediction with recurrent neural networks
S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer · 2015
Cited alongside, same era.

Then

Trust region policy optimization
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz · 2015
Later among the works it cites.
End-to-end training of deep visuomotor policies
S. Levine, C. Finn, T. Darrell, and P. Abbeel · 2016
Later among the works it cites.
Benchmarking deep reinforcement learning for continuous control
Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel · 2016
Later among the works it cites.
Exploration from demonstration for interactive reinforcement learning
K. Subramanian, C. L. Isbell, Jr., and A. L. Thomaz · 2016
Later among the works it cites.
Generalization and exploration via randomized value functions
I. Osband, B. Van Roy, and Z. Wen · 2016
Later among the works it cites.
Automatic goal generation for reinforcement learning agents
Original
D. Held, X. Geng, C. Florensa, and P. Abbeel · 2017
Closest in time.
Data-efficient deep reinforcement learning for dexterous manipulation
I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y. Tassa, T. Erez, and M. Riedmiller · 2017
Closest in time.
Automated Curriculum Learning for Neural Networks
Original
A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu · 2017
Closest in time.
Online Multi-Task Learning Using Biased Sampling
S. Sharma and B. Ravindran · 2017
Closest in time.
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
S. Sukhbaatar, I. Kostrikov, A. Szlam, and R. Fergus · 2017
Closest in time.
Towards generalization and simplicity in continuous control
Original
A. Rajeswaran, K. Lowrey, E. Todorov, and S. Kakade · 2017
Closest in time.
Intrinsic motivation and automatic curricula via asymmetric self-play
Original
S. Sukhbaatar, I. Kostrikov, A. Szlam, and R. Fergus · 2017
Closest in time.
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
Original
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…