2017

Count-Based Exploration with Neural Density Models

Ostrovski, Georg, Bellemare, Marc G., Oord, Aaron van den et al.

Understand

Bellemare et al.

(2016) introduced the notion of a pseudo-count, derived from a density model, to generalize count-based exploration to non-tabular reinforcement learning.
This pseudo-count was used to generate an exploration bonus for a DQN agent and combined with a mixed Monte Carlo update was sufficient to achieve state of the art on the Atari 2600 game Montezuma's Revenge.
We consider two questions left open by their work: First, how important is the quality of the density model for exploration? Second, what role does the Monte Carlo update play in exploration? We answer the first question by demonstrating the use of PixelCNN, an advanced neural density model for images, to supply a pseudo-count.

Built on

Generalization in reinforcement learning: Successful examples using sparse coarse coding
Sutton, Richard S · 1996
Earlier work this paper cites.
An analysis of temporal-difference learning with function approximation
Tsitsiklis, John N. and van Roy, Benjamin · 1997
Earlier work this paper cites.
Reinforcement learning: An introduction
Sutton, Richard S. and Barto, Andrew G · 1998
Earlier work this paper cites.
Catastrophic forgetting in connectionist networks
French, Robert M · 1999
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Oudeyer, Pierre-Yves, Kaplan, Frédéric, and Hafner, Verena V · 2007
Earlier work this paper cites.
An analysis of model-based interval estimation for Markov decision processes
Strehl, Alexander L. and Littman, Michael L · 2008
Earlier work this paper cites.
Near-optimal regret bounds for reinforcement learning
Jaksch, Thomas, Ortner, Ronald, and Auer, Peter · 2010
Earlier work this paper cites.
PILCO: A model-based and data-efficient approach to policy search
Deisenroth, Marc P. and Rasmussen, Carl E · 2011
Earlier work this paper cites.
Efficient bayes-adaptive reinforcement learning using sample-based search
Guez, Arthur, Silver, David, and Dayan, Peter · 2012
Earlier work this paper cites.
Exploration in model-based reinforcement learning by empirically estimating learning progress
Lopes, Manuel, Lang, Tobias, Toussaint, Marc, and Oudeyer, Pierre-Yves · 2012
Earlier work this paper cites.

Similar

RMSProp: divide the gradient by a running average of its recent magnitude
Tieleman, Tijmen and Hinton, Geoffrey · 2012
Cited alongside, same era.
Context tree switching
Veness, Joel, Ng, Kee Siong, Hutter, Marcus, and Bowling, Michael H · 2012
Cited alongside, same era.
The arcade learning environment: An evaluation platform for general agents
Bellemare, Marc G., Naddaf, Yavar, Veness, Joel, and Bowling, Michael · 2013
Cited alongside, same era.
Auto-encoding variational bayes
Kingma, Diederik P. and Welling, Max · 2013
Cited alongside, same era.
Skip context tree switching
Bellemare, Marc, Veness, Joel, and Talvitie, Erik · 2014
Cited alongside, same era.
Generative adversarial nets
Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, Danilo Jimenez, Mohamed, Shakir, and Wierstra, Daan · 2014
Cited alongside, same era.
Draw: A recurrent neural network for image generation
Gregor, Karol, Danihelka, Ivo, Graves, Alex, Rezende, Danilo, and Wierstra, Daan · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, et al · 2015
Cited alongside, same era.
Conditional image generation with PixelCNN decoders
van den Oord, Aaron, Kalchbrenner, Nal, Espeholt, Lasse, Vinyals, Oriol, Graves, Alex, et al
Cited in the paper.
Pixel recurrent neural networks
van den Oord, Aaron, Kalchbrenner, Nal, and Kavukcuoglu, Koray
Cited in the paper.

Then

Unifying count-based exploration and intrinsic motivation
Bellemare, Marc G., Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, Rémi · 2016
Later among the works it cites.
Variational information maximizing exploration
Houthooft, Rein, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter · 2016
Later among the works it cites.
Safe and efficient off-policy reinforcement learning
Munos, Rémi, Stepleton, Tom, Harutyunyan, Anna, and Bellemare, Marc G · 2016
Later among the works it cites.
Generalization and exploration via randomized value functions
Osband, Ian, van Roy, Benjamin, and Wen, Zheng · 2016
Later among the works it cites.
#Exploration: A study of count-based exploration for deep reinforcement learning
Original
Tang, Haoran, Houthooft, Rein, Foote, Davis, Stooke, Adam, Chen, Xi, Duan, Yan, Schulman, John, De Turck, Filip, and Abbeel, Pieter · 2016
Later among the works it cites.
A note on the evaluation of generative models
Theis, Lucas, van den Oord, Aäron, and Bethge, Matthias · 2016
Later among the works it cites.
Deep reinforcement learning with Double Q-learning
van Hasselt, Hado, Guez, Arthur, and Silver, David · 2016
Later among the works it cites.
Dueling network architectures for deep reinforcement learning
Wang, Ziyu, Schaul, Tom, Hessel, Matteo, van Hasselt, Hado, Lanctot, Marc, and de Freitas, Nando · 2016
Later among the works it cites.
The Reactor: A sample-efficient actor-critic architecture
Original
Gruslys, Audrunas, Azar, Mohammad Gheshlaghi, Bellemare, Marc G., and Munos, Rémi · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…