2020

Meta-trained agents implement Bayes-optimal agents

Mikulik, Vladimir, Delétang, Grégoire, McGrath, Tom et al.

Understand

Memory-based meta-learning is a powerful technique to build agents that adapt fast to any task within a target distribution.

A previous theoretical study has argued that this remarkable performance is because the meta-training protocol incentivises agents to behave Bayes-optimally.
We empirically investigate this claim on a number of prediction and bandit tasks.
Inspired by ideas from theoretical computer science, we show that meta-learned and Bayes-optimal agents not only behave alike, but they even share a similar computational structure, in the sense that one agent system can approximately simulate the other.

Built on

A method for synthesizing sequential circuits
George H Mealy · 1955
Earlier work this paper cites.
Applied statistical decision theory
Howard Raiffa and Robert Schlaifer · 1961
Earlier work this paper cites.
Bandit processes and dynamic allocation indices
John C Gittins · 1979
Earlier work this paper cites.
The utility driven dynamic error propagation network
AJ Robinson and Frank Fallside · 1987
Earlier work this paper cites.
Generalization of backpropagation with application to a recurrent gas market model
Paul J Werbos · 1988
Earlier work this paper cites.
Action and planning in embedded agents
Leslie Pack Kaelbling and Stanley J Rosenschein · 1990
Earlier work this paper cites.
Learning a synaptic learning rule
Y Bengio, S Bengio, and J Cloutier · 1991
Earlier work this paper cites.
A situated view of representation and control
Stanley J Rosenschein and Leslie Pack Kaelbling · 1995
Earlier work this paper cites.
Simple principles of metalearning
Juergen Schmidhuber, Jieyu Zhao, and MA Wiering · 1996
Earlier work this paper cites.
Introduction to the theory of computation
Michael Sipser · 1996
Earlier work this paper cites.
Learning to learn: Introduction and overview
Sebastian Thrun and Lorien Pratt · 1998
Earlier work this paper cites.
Models of computation. exploring the power of computing
JE Savage · 1998
Earlier work this paper cites.
Learning to forget: Continual prediction with LSTM
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins · 1999
Earlier work this paper cites.
Learning to learn using gradient descent
Sepp Hochreiter, A Steven Younger, and Peter R Conwell · 2001
Earlier work this paper cites.
A field guide to dynamical recurrent networks
John F Kolen and Stefan C Kremer · 2001
Earlier work this paper cites.
Computational mechanics: Pattern and prediction, structure and simplicity
Cosma Rohilla Shalizi and James P Crutchfield · 2001
Earlier work this paper cites.
Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes
Michael O’Gordon Duff and Andrew Barto · 2002
Earlier work this paper cites.
On the computational power of circuits of spiking neurons
Wolfgang Maass and Henry Markram · 2004
Earlier work this paper cites.
Modern multidimensional scaling: Theory and applications
Ingwer Borg and Patrick JF Groenen · 2005
Earlier work this paper cites.
Approximate bisimulations for nonlinear dynamical systems
Antoine Girard and George J Pappas · 2005
Earlier work this paper cites.
Pattern recognition and machine learning
Christopher M Bishop · 2006
Earlier work this paper cites.
Principles of model checking
Christel Baier and Joost-Pieter Katoen · 2008
Earlier work this paper cites.
Motor task variation induces structural learning
Daniel A Braun, Ad Aertsen, Daniel M Wolpert, and Carsten Mehring · 2009
Earlier work this paper cites.

Similar

Visualizing higher-layer features of a deep network
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent · 2009
Cited alongside, same era.
Structure learning in action
Daniel A Braun, Carsten Mehring, and Daniel M Wolpert · 2010
Cited alongside, same era.
Vision: A computational investigation into the human representation and processing of visual information
David Marr · 2010
Cited alongside, same era.
Bayesian data analysis
Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin · 2013
Cited alongside, same era.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2013
Cited alongside, same era.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Original
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio · 2014
Cited alongside, same era.
Concurrency theory: A historical perspective on coinduction and process calculi
Jos C.M. Baeten and Davide Sangiorgi · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Cited alongside, same era.
Meta-learning with memory-augmented neural networks
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap · 2016
Cited alongside, same era.
Learning to reinforcement learn
Original
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick · 2016
Cited alongside, same era.
Towards a neural statistician
Original
Harrison Edwards and Amos Storkey · 2016
Cited alongside, same era.

Then

Bandit algorithms
Tor Lattimore and Csaba Szepesvári · 2018
Later among the works it cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 2018
Later among the works it cites.
Impala: Scalable distributed deep-RL with importance weighted actor-learner architectures
Original
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al · 2018
Later among the works it cites.
Recasting gradient-based meta-learning as hierarchical bayes
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths · 2018
Later among the works it cites.
Learning finite state representations of recurrent policy networks
Original
Anurag Koul, Sam Greydanus, and Alan Fern · 2018
Later among the works it cites.
Yoshua Bengio, revered architect of AI, has some ideas about what to build next
Eliza Strickland · 2019
Later among the works it cites.
Meta-learning of sequential strategies
Original
Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al · 2019
Later among the works it cites.
Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo · 2019
Later among the works it cites.
From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli · 2019
Later among the works it cites.
Identifying and controlling important neurons in neural machine translation
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass · 2019
Later among the works it cites.
Human-level performance in 3D multiplayer games with population-based reinforcement learning
Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al · 2019
Later among the works it cites.
Meta-learners’ learning dynamics are unlike learners’
Original
Neil C Rabinowitz · 2019
Later among the works it cites.
Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction
Original
Bryan Lim, Stefan Zohren, and Stephen Roberts · 2019
Later among the works it cites.
Metalearned neural memory
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler · 2019
Later among the works it cites.
Learning causal state representations of partially observable environments
Original
Amy Zhang, Zachary C Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello · 2019
Later among the works it cites.
Analysis methods in neural language processing: A survey
Yonatan Belinkov and James Glass · 2019
Later among the works it cites.
Bayesian computation through cortical latent dynamics
Hansem Sohn, Devika Narain, Nicolas Meirhaeghe, and Mehrdad Jazayeri · 2019
Later among the works it cites.
Universality and individuality in neural dynamics across large populations of recurrent networks
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo · 2019
Later among the works it cites.
A theory of learning to infer
Ishita Dasgupta, Eric Schulz, Joshua B. Tenenbaum, and Samuel J. Gershman · 2019
Later among the works it cites.
How recurrent networks implement contextual processing in sentiment analysis
Original
Niru Maheswaranathan and David Sussillo · 2020
Closest in time.
An overview of early vision in InceptionV1
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter · 2020
Closest in time.
Rapid learning or feature reuse? Towards understanding the effectiveness of MAML
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals · 2020
Closest in time.
Understanding the role of individual units in a deep neural network
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…