2020

Deep Reinforcement and InfoMax Learning

Mazoure, Bogdan, Combes, Remi Tachet des, Doan, Thang et al.

Understand

We begin with the hypothesis that a model-free agent whose representations are predictive of properties of future states (beyond expected rewards) will be more capable of solving and adapting to new RL problems.

To test that hypothesis, we introduce an objective based on Deep InfoMax (DIM) which trains the agent to predict the future by maximizing the mutual information between its internal representation of successive timesteps.
We test our approach in several synthetic settings, where it successfully learns representations that are predictive of the future.
Finally, we augment C51, a strong RL baseline, with our temporal DIM objective and demonstrate improved performance on a continual learning task and on the recently introduced Procgen environment.

Built on

Dream to control: Learning behaviors by latent imagination
Original
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi · 1912
Earlier work this paper cites.
A markovian decision process
Richard Bellman · 1957
Earlier work this paper cites.
Reinforcement Learning: An Introduction
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Learning to learn: Introduction and overview
Sebastian Thrun and Lorien Pratt · 1998
Earlier work this paper cites.
The psychology and neuroscience of forgetting
John T Wixted · 2004
Earlier work this paper cites.
Nonhomogeneous geometric distributions with relations to birth and death processes
Marvin Mandelbaum, Myron Hlynka, and Percy H Brill · 2007
Earlier work this paper cites.
A tutorial on spectral clustering
Ulrike Von Luxburg · 2007
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Michael Gutmann and Aapo Hyvärinen · 2010
Earlier work this paper cites.
Comparison of co-expression measures: mutual information, correlation, and model based indices
Lin Song, Peter Langfelder, and Steve Horvath · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Earlier work this paper cites.
Markov decision processes: discrete stochastic dynamic programming
Martin L Puterman · 2014
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Earlier work this paper cites.
Variational information maximisation for intrinsically motivated reinforcement learning
Shakir Mohamed and Danilo Jimenez Rezende · 2015
Earlier work this paper cites.
Unsupervised feature extraction by time-contrastive learning and nonlinear ica
Aapo Hyvarinen and Hiroshi Morioka · 2016
Earlier work this paper cites.
Reinforcement learning with unsupervised auxiliary tasks
Original
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Deepmind lab, 2016
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen · 2016
Earlier work this paper cites.
Deep learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2017
Earlier work this paper cites.

Similar

Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
A distributional perspective on reinforcement learning
Marc G Bellemare, Will Dabney, and Rémi Munos · 2017
Cited alongside, same era.
Markov chains and mixing times , volume 107
David A Levin and Yuval Peres · 2017
Cited alongside, same era.
World models
Original
David Ha and Jürgen Schmidhuber · 2018
Cited alongside, same era.
Temporal difference models: Model-free deep rl for model-based control
Original
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine · 2018
Cited alongside, same era.
Generalization and regularization in dqn, 2018
Jesse Farebrother, Marlos C. Machado, and Michael Bowling · 2018
Cited alongside, same era.
Learning invariances for policy generalization
Original
Remi Tachet des Combes, Philip Bachman, and Harm van Seijen · 2018
Cited alongside, same era.
Learning deep representations by mutual information estimation and maximization
Original
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio · 2018
Cited alongside, same era.
Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting
Original
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins · 2018
Cited alongside, same era.
Continual reinforcement learning with complex synapses
Original
Christos Kaplanis, Murray Shanahan, and Claudia Clopath · 2018
Cited alongside, same era.
Unicorn: Continual learning with a universal, off-policy agent
Original
Daniel J Mankowitz, Augustin Žídek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, and Tom Schaul · 2018
Cited alongside, same era.

Then

Leveraging procedural generation to benchmark reinforcement learning, 2019
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman · 2019
Later among the works it cites.
Multi-task deep reinforcement learning with popart
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt · 2019
Later among the works it cites.
Sharing knowledge in multi-task deep reinforcement learning
Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters · 2019
Later among the works it cites.
Deepmdp: Learning continuous latent space models for representation learning
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare · 2019
Later among the works it cites.
Mastering atari, go, chess and shogi by planning with a learned model, 2019
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver · 2019
Later among the works it cites.
Data-efficient image recognition with contrastive predictive coding
Original
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord · 2019
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Original
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2019
Later among the works it cites.
Contrastive multiview coding
Original
Yonglong Tian, Dilip Krishnan, and Phillip Isola · 2019
Later among the works it cites.
On variational bounds of mutual information
Original
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker · 2019
Later among the works it cites.
Emi: Exploration with mutual information
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song · 2019
Later among the works it cites.
Kinematic state abstraction and provably efficient rich-observation reinforcement learning, 2019
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford · 2019
Later among the works it cites.
Distributional reinforcement learning with linear function approximation
Original
Marc G Bellemare, Nicolas Le Roux, Pablo Samuel Castro, and Subhodeep Moitra · 2019
Later among the works it cites.
A theoretical analysis of catastrophic forgetting through the ntk overlap matrix
Original
Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier · 2020
Closest in time.
A simple framework for contrastive learning of visual representations
Original
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton · 2020
Closest in time.
Curl: Contrastive unsupervised representations for reinforcement learning
Original
Aravind Srinivas, Michael Laskin, and Pieter Abbeel · 2020
Closest in time.
Provably efficient reconstruction of policy networks
Original
Bogdan Mazoure, Thang Doan, Tianyu Li, Vladimir Makarenkov, Joelle Pineau, Doina Precup, and Guillaume Rabusseau · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…