2017

Learning to Compose Skills

Sahni, Himanshu, Kumar, Saurabh, Tejani, Farhan et al.

Understand

We present a differentiable framework capable of learning a wide variety of compositions of simple policies that we call skills.

By recursively composing skills with themselves, we can create hierarchies that display complex behavior.
Skill networks are trained to generate skill-state embeddings that are provided as inputs to a trainable composition function, which in turn outputs a policy for the overall task.
Our experiments on an environment consisting of multiple collect and evade tasks show that this architecture is able to quickly build complex skills from simpler ones.

The temporal logic of reactive and concurrent systems
A. Pnueli and Z. Manna · 1992
Earlier work this paper cites.
Behavior coordination for a mobile robot using modular reinforcement learning
E. Uchibe, M. Asada, and K. Hosoda · 1996
Earlier work this paper cites.
Introduction to reinforcement learning
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
R. S. Sutton, D. Precup, and S. Singh · 1999
Earlier work this paper cites.
Hierarchical reinforcement learning with the maxq value function decomposition
T. G. Dietterich · 2000
Earlier work this paper cites.
Hierarchical reinforcement learning with the maxq value function decomposition
T. G. Dietterich · 2000
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour · 2000
Earlier work this paper cites.
Recent advances in hierarchical reinforcement learning
A. G. Barto and S. Mahadevan · 2003
Earlier work this paper cites.
Inter-module credit assignment in modular reinforcement learning
K. Samejima, K. Doya, and M. Kawato · 2003
Earlier work this paper cites.

On the difficulty of modular reinforcement learning for real-world partial programming
S. Bhat, C. L. Isbell, and M. Mateas · 2006
Cited alongside, same era.
Building portable options: Skill transfer in reinforcement learning
G. Konidaris and A. G. Barto · 2007
Cited alongside, same era.
Principles of model checking
C. Baier, J.-P. Katoen, and K. G. Larsen · 2008
Cited alongside, same era.
Robot learning from demonstration by constructing skill trees
G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto · 2012
Cited alongside, same era.
Human-level control through deep reinforcement learning
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al · 2015
Cited alongside, same era.
Modular multitask reinforcement learning with policy sketches
Original
J. Andreas, D. Klein, and S. Levine · 2016
Cited alongside, same era.
Constructing abstraction hierarchies using a skill-symbol loop
G. Konidaris · 2016
Cited alongside, same era.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation
T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. Tenenbaum · 2016
Cited alongside, same era.

Asynchronous methods for deep reinforcement learning
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu · 2016
Later among the works it cites.
The option-critic architecture
P.-L. Bacon, J. Harb, and D. Precup · 2017
Closest in time.
Meta learning shared hierarchies
Original
K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman · 2017
Closest in time.
Learning a hierarchy
P. C. P. A. Kevin Frans, Jonathan Ho and J. Schulman · 2017
Closest in time.
Environment-independent task specifications via gltl
Original
M. L. Littman, U. Topcu, J. Fu, C. Isbell, M. Wen, and J. MacGlashan · 2017
Closest in time.
Zero-shot task generalization with multi-task deep reinforcement learning
Original
J. Oh, S. Singh, H. Lee, and P. Kohli · 2017
Closest in time.
Hybrid reward architecture for reinforcement learning
Original
H. van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang · 2017
Closest in time.
Feudal networks for hierarchical reinforcement learning
Original
A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…