2015

Memory-based control with recurrent neural networks

Heess, Nicolas, Hunt, Jonathan J, Lillicrap, Timothy P et al.

Understand

Partially observed control problems are a challenging aspect of reinforcement learning.

We extend two related, model-free algorithms for continuous control -- deterministic policy gradient and stochastic value gradient -- to solve partially observed domains using recurrent neural networks trained with backpropagation through time.
We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements.
These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps.

Built on

Reinforcement learning with hidden states
L.-J. Lin and T. M. Mitchell · 1993
Earlier work this paper cites.
Learning without state-estimation in partially observable markovian decision processes
S. P. Singh · 1994
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Planning and acting in partially observable stochastic domains
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra · 1998
Earlier work this paper cites.
Learning policies with external memory
L. Peshkin, N. Meuleau, and L. P. Kaelbling · 1999
Earlier work this paper cites.
Applications of the morris water maze in the study of learning and memory
R. DHooge and P. P. De Deyn · 2001
Earlier work this paper cites.
Reinforcement learning with long short-term memory
B. Bakker · 2002
Earlier work this paper cites.
Reinforcement learning for factored markov decision processes
B. Sallans · 2002
Earlier work this paper cites.
Probabilistic robotics
S. Thrun, W. Burgard, and D. Fox · 2005
Earlier work this paper cites.
Solving deep memory pomdps with recurrent policy gradients
D. Wierstra, A. Förster, J. Peters, and J. Schmidhuber · 2007
Earlier work this paper cites.
Policy gradient critics
D. Wierstra and J. Schmidhuber · 2007
Earlier work this paper cites.

Similar

Reinforcement learning and adaptive dynamic programming for feedback control
F. L. Lewis and D. Vrabie · 2009
Cited alongside, same era.
Contextual behaviors and internal representations acquired by reinforcement learning with a recurrent neural network in a continuous state and action space task
H. Utsunomiya and K. Shibata · 2009
Cited alongside, same era.
Reinforcement learning in feedback control
R. Hafner and M. Riedmiller · 2011
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Cited alongside, same era.
Mujoco: A physics engine for model-based control
E. Todorov, T. Erez, and Y. Tassa · 2012
Cited alongside, same era.
Auto-encoding variational bayes
Original
D. P. Kingma and M. Welling · 2013
Cited alongside, same era.
A survey of point-based pomdp solvers
G. Shani, J. Pineau, and R. Kaplow · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Cited alongside, same era.
Cnn features off-the-shelf: an astounding baseline for recognition
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
D. J. Rezende, S. Mohamed, and D. Wierstra · 2014
Cited alongside, same era.
Deterministic policy gradient algorithms
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller · 2014
Cited alongside, same era.

Then

Bias in natural actor-critic algorithms
P. Thomas · 2014
Later among the works it cites.
Compatible value gradients for reinforcement learning of continuous deep policies
Original
D. Balduzzi and M. Ghifary · 2015
Closest in time.
Deep recurrent q-learning for partially observable mdps
Original
M. Hausknecht and P. Stone · 2015
Closest in time.
Learning continuous control policies by stochastic value gradients
N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa · 2015
Closest in time.
End-to-end training of deep visuomotor policies
Original
S. Levine, C. Finn, T. Darrell, and P. Abbeel · 2015
Closest in time.
Continuous control with deep reinforcement learning
Original
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra · 2015
Closest in time.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al · 2015
Closest in time.
Trust region policy optimization
J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz · 2015
Closest in time.
High-dimensional continuous control using generalized advantage estimation
Original
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel · 2015
Closest in time.
Policy learning with continuous memory states for partially observed robotic control
Original
M. Zhang, S. Levine, Z. McCarthy, C. Finn, and P. Abbeel · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…