2019

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

Barreto, André, Borsa, Diana, Quan, John et al.

Understand

The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach.

Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills.
In this paper we extend the SFs & GPI framework in two ways.
One of the basic assumptions underlying the original formulation of SFs & GPI is that rewards for all tasks of interest can be computed as linear combinations of a fixed set of features.

Built on

Parallel distributed processing: Explorations in the microstructure of cognition
Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1986
Earlier work this paper cites.
Learning from Delayed Rewards
Watkins, C · 1989
Earlier work this paper cites.
Q-learning
Watkins, C. and Dayan, P · 1992
Earlier work this paper cites.
Improving generalization for temporal difference learning: The successor representation
Dayan, P · 1993
Earlier work this paper cites.
Markov Decision Processes—Discrete Stochastic Dynamic Programming
Puterman, M. L · 1994
Earlier work this paper cites.
Neuro-Dynamic Programming
Bertsekas, D. P. and Tsitsiklis, J. N · 1996
Earlier work this paper cites.
Is learning the N-th thing any easier than learning the first?
Thrun, S · 1996
Earlier work this paper cites.
Multitask learning
Caruana, R · 1997
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Reinforcement Learning: An Introduction
Sutton, R. S. and Barto, A. G · 1998
Earlier work this paper cites.
A model of inductive bias learning
Baxter, J · 2000
Earlier work this paper cites.
Transfer learning for reinforcement learning domains: A survey
Taylor, M. E. and Stone, P · 2009
Earlier work this paper cites.

Similar

Algorithms for Reinforcement Learning
Szepesvári, C · 2010
Cited alongside, same era.
Transfer in Reinforcement Learning: A Framework and a Survey , pp. 143–173
Lazaric, A · 2012
Cited alongside, same era.
Heads-up limit hold’em poker is solved
Bowling, M., Burch, N., Johanson, M., and Tammelin, O · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Cited alongside, same era.
Deepmind lab
Original
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al · 2016
Cited alongside, same era.
Learning modular neural network policies for multi-task and multi-robot transfer
Original
Devin, C., Gupta, A., Darrell, T., Abbeel, P., and Levine, S · 2016
Cited alongside, same era.
Deep Learning
Goodfellow, I., Bengio, Y., and Courville, A · 2016
Cited alongside, same era.
Learning and transfer of modulated locomotor controllers
Original
Heess, N., Wayne, G., Tassa, Y., Lillicrap, T. P., Riedmiller, M. A., and Silver, D · 2016
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
Original
Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R · 2016
Cited alongside, same era.
Deep successor reinforcement learning
Original
Kulkarni, T. D., Saeedi, A., Gautam, S., and Gershman, S. J · 2016
Cited alongside, same era.
Progressive neural networks
Original
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D · 2016
Cited alongside, same era.

Then

Deep reinforcement learning with successor features for navigation across similar environments
Original
Zhang, J., Springenberg, J. T., Boedecker, J., and Burgard, W · 2016
Later among the works it cites.
The option-critic architecture
Bacon, P., Harb, J., and Precup, D · 2017
Later among the works it cites.
Successor features for transfer in reinforcement learning
Barreto, A., Dabney, W., Munos, R., Hunt, J., Schaul, T., van Hasselt, H., and Silver, D · 2017
Later among the works it cites.
Policy transfer via modularity and reward guiding
Clavera, I., Held, D., and Abbeel, P · 2017
Later among the works it cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Original
Finn, C., Abbeel, P., and Levine, S · 2017
Later among the works it cites.
Meta learning shared hierarchies
Original
Frans, K., Ho, J., Chen, X., Abbeel, P., and Schulman, J · 2017
Later among the works it cites.
Emergence of locomotion behaviours in rich environments
Original
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., Riedmiller, M. A., and Silver, D · 2017
Later among the works it cites.
Zero-shot task generalization with multi-task deep reinforcement learning
Original
Oh, J., Singh, S. P., Lee, H., and Kohli, P · 2017
Later among the works it cites.
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D · 2017
Later among the works it cites.
Distral: Robust multitask reinforcement learning
Teh, Y. W., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R · 2017
Later among the works it cites.
FeUdal networks for hierarchical reinforcement learning
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K · 2017
Later among the works it cites.
IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures
Original
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…