2019

Meta-learning of Sequential Strategies

Ortega, Pedro A., Wang, Jane X., Rowland, Mark et al.

Understand

In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class.

Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains.
To do so, we present basic algorithmic templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model that allowed them to efficiently exploit task structure.
Furthermore, we recast memory-based meta-learning within a Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data, where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics.

Built on

"general intelligence," objectively determined and measured
Charles Spearman · 1904
Earlier work this paper cites.
On the likelihood that one unknown probability exceeds another in view of the evidence of two samples
William R Thompson · 1933
Earlier work this paper cites.
On distributions admitting a sufficient statistic
Bernard Osgood Koopman · 1936
Earlier work this paper cites.
Mental tests used in genetic, the performance of related indiviuals on tests mainly educative and mainly reproductive
John C Raven · 1936
Earlier work this paper cites.
The theory of dynamic programming
Richard Bellman · 1954
Earlier work this paper cites.
A formal theory of inductive inference. part i
Ray J Solomonoff · 1964
Earlier work this paper cites.
Algebraic theory of machines, languages, and semigroups
Kenneth Krohn, John L Rhodes, and Michael A Arbib · 1968
Earlier work this paper cites.
The foundations of statistics
Leonard J Savage · 1972
Earlier work this paper cites.
Optimization by simulated annealing
Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi · 1983
Earlier work this paper cites.
Convergence and finite-time behavior of simulated annealing
Debasis Mitra, Fabio Romeo, and Alberto Sangiovanni-Vincentelli · 1986
Earlier work this paper cites.
Sufficiency as statistical symmetry
Persi Diaconis · 1988
Earlier work this paper cites.
Finding structure in time
Jeffrey L Elman · 1990
Earlier work this paper cites.
Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments
Jürgen Schmidhuber · 1990
Earlier work this paper cites.
Learning a synaptic learning rule
Yoshua Bengio, Samy Bengio, Jocelyn Cloutier, and Jan Gecsei · 1991
Earlier work this paper cites.
Fool’s gold: Extracting finite state machines from recurrent network dynamics
John F Kolen · 1994
Earlier work this paper cites.
Markov games as a framework for multi-agent reinforcement learning
Michael L Littman · 1994
Earlier work this paper cites.
Quantal response equilibria for normal form games
Richard D McKelvey and Thomas R Palfrey · 1995
Earlier work this paper cites.
Simple principles of metalearning
Jurgen Schmidhuber, Jieyu Zhao, and Marco Wiering · 1996
Earlier work this paper cites.
Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography, 1997
Linda S Gottfredson · 1997
Earlier work this paper cites.
Serial order: A parallel distributed processing approach
Michael I Jordan · 1997
Earlier work this paper cites.
An introduction to Kolmogorov complexity and its applications , volume 34
Paul MB Vitanyi and Ming Li · 1997
Earlier work this paper cites.
Theoretical models of learning to learn
Jonathan Baxter · 1998
Earlier work this paper cites.
Quantal response equilibria for extensive form games
Richard D McKelvey and Thomas R Palfrey · 1998
Earlier work this paper cites.
A comparison of simulated annealing cooling strategies
Yaghout Nourani and Bjarne Andresen · 1998
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton, Andrew G Barto, Francis Bach, et al · 1998
Earlier work this paper cites.
Learning to learn: Introduction and overview
Sebastian Thrun and Lorien Pratt · 1998
Earlier work this paper cites.
Prequential probability: Principles and properties
A Philip Dawid, Vladimir G Vovk, et al · 1999
Earlier work this paper cites.
A model of inductive bias learning
Jonathan Baxter · 2000
Earlier work this paper cites.
Beyond the turing test
Jose Hernandez-Orallo · 2000
Earlier work this paper cites.

Similar

Learning to learn using gradient descent
Sepp Hochreiter, A Steven Younger, and Peter R Conwell · 2001
Cited alongside, same era.
A field guide to dynamical recurrent networks
John F Kolen and Stefan C Kremer · 2001
Cited alongside, same era.
Probability theory: The logic of science
Edwin T Jaynes · 2003
Cited alongside, same era.
Universal artificial intelligence: Sequential decisions based on algorithmic probability
Marcus Hutter · 2004
Cited alongside, same era.
Prediction, learning, and games
Nicolo Cesa-Bianchi and Gábor Lugosi · 2006
Cited alongside, same era.
Introduction to the Theory of Computation , volume 2
Michael Sipser · 2006
Cited alongside, same era.
Universal intelligence: A definition of machine intelligence
Shane Legg and Marcus Hutter · 2007
Cited alongside, same era.
Neuro-dynamic programming
Dimitri P Bertsekas · 2008
Cited alongside, same era.
Robustness
Lars Peter Hansen and Thomas J Sargent · 2008
Cited alongside, same era.
Causality
Judea Pearl · 2009
Cited alongside, same era.
A minimum relative entropy principle for learning and acting
Pedro A Ortega and Daniel A Braun · 2010
Cited alongside, same era.
Monte-carlo planning in large pomdps
David Silver and Joel Veness · 2010
Cited alongside, same era.

Then

Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Posterior sampling for reinforcement learning without episodes
Original
Ian Osband and Benjamin Van Roy · 2016
Later among the works it cites.
Optimization as a model for few-shot learning
Sachin Ravi and Hugo Larochelle · 2016
Later among the works it cites.
Deep amortized inference for probabilistic programs
Original
Daniel Ritchie, Paul Horsfall, and Noah D Goodman · 2016
Later among the works it cites.
Meta-learning with memory-augmented neural networks
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap · 2016
Later among the works it cites.
The predictron: End-to-end learning and planning
Original
David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al · 2016
Later among the works it cites.
Matching networks for one shot learning
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al · 2016
Later among the works it cites.
Learning to reinforcement learn
Original
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick · 2016
Later among the works it cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Original
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Later among the works it cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell · 2017
Later among the works it cites.
Efficient probabilistic inference in generic neural networks trained with non-probabilistic feedback
A Emin Orhan and Wei Ji Ma · 2017
Later among the works it cites.
Learning unknown markov decision processes: A thompson sampling approach
Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain · 2017
Later among the works it cites.
Prototypical networks for few-shot learning
Jake Snell, Kevin Swersky, and Richard Zemel · 2017
Later among the works it cites.
Robust imitation of diverse behaviors
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess · 2017
Later among the works it cites.
Learned optimizers that scale and generalize
Original
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein · 2017
Later among the works it cites.
Conditional neural processes
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami · 2018
Later among the works it cites.
Meta-learning probabilistic inference for prediction
Original
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner · 2018
Later among the works it cites.
Recasting gradient-based meta-learning as hierarchical bayes
Original
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths · 2018
Later among the works it cites.
Continual reinforcement learning with complex synapses
Christos Kaplanis, Murray Shanahan, and Claudia Clopath · 2018
Later among the works it cites.
A simple neural attentive meta-learner
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel · 2018
Later among the works it cites.
Reptile: a scalable metalearning algorithm
Original
Alex Nichol and John Schulman · 2018
Later among the works it cites.
Modeling friends and foes
Original
Pedro A Ortega and Shane Legg · 2018
Later among the works it cites.
Building safe artificial intelligence: specification, robustness, and assurance
Pedro A Ortega, Vishal Maini, and the DeepMind Safety Team · 2018
Later among the works it cites.
Continual lifelong learning with neural networks: A review, 2018
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter · 2018
Later among the works it cites.
A tutorial on thompson sampling
Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al · 2018
Later among the works it cites.
Meta-gradient reinforcement learning
Zhongwen Xu, Hado van Hasselt, and David Silver · 2018
Later among the works it cites.
Caml: Fast context adaptation via meta-learning
Original
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson · 2018
Later among the works it cites.
Causal reasoning from meta-reinforcement learning
Original
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson · 2019
Closest in time.
Deep online learning via meta-learning: Continual adaptation for model-based RL
Anusha Nagabandi, Chelsea Finn, and Sergey Levine · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…