2017

Eigenoption Discovery through the Deep Successor Representation

Machado, Marlos C., Rosenbaum, Clemens, Guo, Xiaoxiao et al.

Understand

Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning.

However, autonomously learning effective sets of options is still a major challenge in the field.
In this paper we focus on the recently introduced idea of using representation learning methods to guide the option discovery process.
Specifically, we look at eigenoptions, options obtained from representations that encode diffusive information flow in the environment.

Built on

Learning to Predict by the Methods of Temporal Differences
Richard S. Sutton · 1988
Earlier work this paper cites.
Technical Note: Q-Learning
Christopher J. C. H. Watkins and Peter Dayan · 1992
Earlier work this paper cites.
Improving Generalization for Temporal Difference Learning: The Successor Representation
Peter Dayan · 1993
Earlier work this paper cites.
Temporal Difference Learning and TD-Gammon
Gerald Tesauro · 1995
Earlier work this paper cites.
Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning
Richard S. Sutton, Doina Precup, and Satinder P. Singh · 1999
Earlier work this paper cites.
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
Amy McGovern and Andrew G. Barto · 2001
Earlier work this paper cites.
Slow Feature Analysis: Unsupervised Learning of Invariances
Laurenz Wiskott and Terrence J. Sejnowski · 2002
Earlier work this paper cites.
Using Relative Novelty to Identify Useful Temporal Abstractions in Reinforcement Learning
Özgür Şimsek and Andrew G. Barto · 2004
Earlier work this paper cites.
Proto-value Functions: Developmental Reinforcement Learning
Sridhar Mahadevan · 2005
Earlier work this paper cites.
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes
Sridhar Mahadevan and Mauro Maggioni · 2007
Earlier work this paper cites.
Dual Representations for Dynamic Programming and Reinforcement Learning
T. Wang, M. Bowling, and D. Schuurmans · 2007
Earlier work this paper cites.

Similar

The Utility of Temporal Abstraction in Reinforcement Learning
Nicholas K. Jong, Todd Hester, and Peter Stone · 2008
Cited alongside, same era.
Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining
George Konidaris and Andrew G. Barto · 2009
Cited alongside, same era.
On the Relation of Slow Feature Analysis and Laplacian Eigenmaps
Henning Sprekeler · 2011
Cited alongside, same era.
Investigating Contingency Awareness Using Atari 2600 Games
Marc G. Bellemare, Joel Veness, and Michael Bowling · 2012
Cited alongside, same era.
The Arcade Learning Environment: An Evaluation Platform for General Agents
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Cited alongside, same era.
PAC-inspired Option Discovery in Lifelong Reinforcement Learning
Emma Brunskill and Lihong Li · 2014
Cited alongside, same era.
Optimal Behavioral Hierarchy
Alec Solway, Carlos Diuk, Natalia Córdova, Debbie Yee, Andrew G. Barto, Yael Niv, and Matthew M. Botvinick · 2014
Cited alongside, same era.
Design Principles of the Hippocampal Cognitive Map
Kimberly L. Stachenfeld, Matthew Botvinick, and Samuel J. Gershman · 2014
Cited alongside, same era.
Human-level Control through Deep Reinforcement Learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Cited alongside, same era.
Action-Conditional Video Prediction using Deep Networks in Atari Games
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder P. Singh · 2015
Cited alongside, same era.
Probabilistic Inference for Determining Options in Reinforcement Learning
Christian Daniel, Herke van Hoof, Jan Peters, and Gerhard Neumann · 2016
Cited alongside, same era.
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum
Cited in the paper.

Then

Adaptive Skills Adaptive Partitions (ASAP)
Daniel J. Mankowitz, Timothy Arthur Mann, and Shie Mannor · 2016
Later among the works it cites.
The Option-Critic Architecture
Pierre-Luc Bacon, Jean Harb, and Doina Precup · 2017
Closest in time.
Successor Features for Transfer in Reinforcement Learning
André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, David Silver, and Hado van Hasselt · 2017
Closest in time.
Stochastic Neural Networks for Hierarchical Reinforcement Learning
Carlos Florensa, Yan Duan, and Pieter Abbeel · 2017
Closest in time.
Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
Zhaohan Daniel Guo, Philip S. Thomas, and Emma Brunskill · 2017
Closest in time.
Reinforcement Learning with Unsupervised Auxiliary Tasks
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu · 2017
Closest in time.
Continual Curiosity-driven Skill Acquisition from High-Dimensional Video Inputs for Humanoid Robots
Varun Raj Kompella, Marijn F. Stollenga, Matthew D. Luciw, and Jürgen Schmidhuber · 2017
Closest in time.
A Laplacian Framework for Option Discovery in Reinforcement Learning
Marlos C. Machado, Marc G. Bellemare, and Michael Bowling · 2017
Closest in time.
The Hippocampus as a Predictive Map
Kimberly L. Stachenfeld, Matthew M Botvinick, and Samuel J. Gershman · 2017
Closest in time.
FeUdal Networks for Hierarchical Reinforcement Learning
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu · 2017
Closest in time.
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…