2016

Model-Free Episodic Control

Blundell, Charles, Uria, Benigno, Pritzel, Alexander et al.

Understand

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance.

Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery.
In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory.
Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks.

Built on

Simple memory: A theory for archicortex
D Marr · 1971
Earlier work this paper cites.
Neural networks and physical systems with emergent collective computational abilities
John J Hopfield · 1982
Earlier work this paper cites.
Place navigation impaired in rats with hippocampal lesions
RGM Morris, P Garrud, and JNP Rawlinst · 1982
Earlier work this paper cites.
A behavioural analysis of spatial localization following electrolytic, kainate-or colchicine-induced damage to the hippocampal formation in the rat
Robert J Sutherland, Ian Q Whishaw, and Bob Kolb · 1983
Earlier work this paper cites.
Extensions of lipschitz mappings into a hilbert space
William B Johnson and Joram Lindenstrauss · 1984
Earlier work this paper cites.
Hippocampal synaptic enhancement and information storage within a distributed memory system
Bruce L McNaughton and Richard GM Morris · 1987
Earlier work this paper cites.
Configural association theory: The role of the hippocampal formation in learning, memory, and amnesia
Robert J Sutherland and Jerry W Rudy · 1989
Earlier work this paper cites.
Long-lasting perceptual priming and semantic learning in amnesia: a case experiment
Original
Endel Tulving, CA Hayman, and Carol A Macdonald · 1991
Earlier work this paper cites.
Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans
Larry R Squire · 1992
Earlier work this paper cites.
Perirhinal and parahippocampal cortices of the macaque monkey: cortical afferents
Wendy L Suzuki and David G Amaral · 1994
Earlier work this paper cites.
Computational analysis of the role of the hippocampus in memory
Alessandro Treves and Edmund T Rolls · 1994
Earlier work this paper cites.
Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory
James L McClelland, Bruce L McNaughton, and Randall C O’Reilly · 1995
Earlier work this paper cites.
Considerations arising from a complementary learning systems perspective on hippocampus and neocortex
James L McClelland and Nigel H Goddard · 1996
Earlier work this paper cites.
Episodic-like memory during cache recovery by scrub jays
Nicola S Clayton and Anthony Dickinson · 1998
Earlier work this paper cites.

Similar

Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Cited alongside, same era.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 1998
Cited alongside, same era.
Recognition memory: what are the roles of the perirhinal cortex and hippocampus?
Malcolm W Brown and John P Aggleton · 2001
Cited alongside, same era.
Requirement for hippocampal ca3 nmda receptors in associative memory recall
Kazu Nakazawa, Michael C Quirk, Raymond A Chitwood, Masahiko Watanabe, Mark F Yeckel, Linus D Sun, Akira Kato, Candice A Carr, Daniel Johnston, Matthew A Wilson, et al · 2002
Cited alongside, same era.
Information theory, inference and learning algorithms
David JC MacKay · 2003
Cited alongside, same era.
Modeling hippocampal and neocortical contributions to recognition memory: a complementary-learning-systems approach
Kenneth A Norman and Randall C O’Reilly · 2003
Cited alongside, same era.
Memory systems of the brain: a brief history and current perspective
Larry R Squire · 2004
Cited alongside, same era.
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control
Nathaniel D Daw, Yael Niv, and Peter Dayan · 2005
Cited alongside, same era.
The hippocampus book
Per Andersen, Richard Morris, David Amaral, Tim Bliss, and John OKeefe · 2006
Cited alongside, same era.
Reverse replay of behavioural sequences in hippocampal place cells during the awake state
David J Foster and Matthew A Wilson · 2006
Cited alongside, same era.
Hippocampal contributions to control: The third way
M. Lengyel and P. Dayan · 2007
Cited alongside, same era.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton · 2010
Cited alongside, same era.

Then

Decay happens: the role of active forgetting in memory
Oliver Hardt, Karim Nader, and Lynn Nadel · 2013
Later among the works it cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Later among the works it cites.
The curse of planning dissecting multiple reinforcement-learning systems by taxing the central executive
A Ross Otto, Samuel J Gershman, Arthur B Markman, and Nathaniel D Daw · 2013
Later among the works it cites.
Semi-supervised learning with deep generative models
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling · 2014
Later among the works it cites.
Stochastic backpropagation and approximate inference in deep generative models
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra · 2014
Later among the works it cites.
Learning to generate chairs with convolutional neural networks
Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox · 2015
Later among the works it cites.
Approximate hubel-wiesel modules and the data structures of neural computation
Original
Joel Z. Leibo, Julien Cornebise, Sergio Gomez, and Demis Hassabis · 2015
Later among the works it cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Later among the works it cites.
Action-conditional video prediction using deep networks in atari games
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh · 2015
Later among the works it cites.
Prioritized experience replay
Original
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver · 2015
Later among the works it cites.
Building machines that learn and think like people
Original
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman · 2016
Closest in time.
Asynchronous methods for deep reinforcement learning
Original
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Closest in time.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…