2017

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Foerster, Jakob, Nardelli, Nantas, Farquhar, Gregory et al.

Understand

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems.

However, existing multi-agent RL methods typically scale poorly in the problem size.
Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting.
A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies.

Built on

Learning from delayed rewards
Watkins, Christopher John Cornish Hellaby · 1989
Earlier work this paper cites.
Multi-agent reinforcement learning: Independent vs. cooperative agents
Tan, Ming · 1993
Earlier work this paper cites.
Long short-term memory
Hochreiter, Sepp and Schmidhuber, Jürgen · 1997
Earlier work this paper cites.
Using communication to reduce locality in distributed multiagent learning
Mataric, Maja J · 1998
Earlier work this paper cites.
Reinforcement learning: An introduction , volume 1
Sutton, Richard S and Barto, Andrew G · 1998
Earlier work this paper cites.
An algorithm for distributed reinforcement learning in cooperative multi-agent systems
Lauer, Martin and Riedmiller, Martin · 2000
Earlier work this paper cites.
Hierarchical multi-agent reinforcement learning
Makar, Rajbala, Mahadevan, Sridhar, and Ghavamzadeh, Mohammad · 2001
Earlier work this paper cites.
Extending q-learning to general adaptive multi-agent systems
Tesauro, Gerald · 2003
Earlier work this paper cites.
Monte carlo statistical methods springer
Robert, CP and Casella, G · 2004
Earlier work this paper cites.
Multiagent reinforcement learning for multi-robot systems: A survey
Yang, Erfu and Gu, Dongbing · 2004
Earlier work this paper cites.
The packet-world: A test bed for investigating situated multi-agent systems
Weyns, Danny, Helleboogh, Alexander, and Holvoet, Tom · 2005
Earlier work this paper cites.
Dealing with non-stationary environments using context detection
Da Silva, Bruno C, Basso, Eduardo W, Bazzan, Ana LC, and Engel, Paulo M · 2006
Earlier work this paper cites.
Collaborative multiagent reinforcement learning by payoff propagation
Kok, Jelle R and Vlassis, Nikos · 2006
Earlier work this paper cites.

Similar

Awesome: A general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents
Conitzer, Vincent and Sandholm, Tuomas · 2007
Cited alongside, same era.
A comprehensive survey of multiagent reinforcement learning
Busoniu, Lucian, Babuska, Robert, and De Schutter, Bart · 2008
Cited alongside, same era.
Multiagent reinforcement learning for urban traffic control using coordination graphs
Kuyer, Lior, Whiteson, Shimon, Bakker, Bram, and Vlassis, Nikos · 2008
Cited alongside, same era.
Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations
Shoham, Y. and Leyton-Brown, K · 2009
Cited alongside, same era.
Torch7: A matlab-like environment for machine learning
Collobert, R., Kavukcuoglu, K., and Farabet, C · 2011
Cited alongside, same era.
Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems
Matignon, Laetitia, Laurent, Guillaume J, and Le Fort-Piat, Nadine · 2012
Cited alongside, same era.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Chung, Junyoung, Gulcehre, Caglar, Cho, KyungHyun, and Bengio, Yoshua · 2014
Cited alongside, same era.
Empirically evaluating multiagent learning algorithms
Zawadzki, E., Lipson, A., and Leyton-Brown, K · 2014
Cited alongside, same era.
Deep recurrent q-learning for partially observable mdps
Original
Hausknecht, Matthew and Stone, Peter · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al · 2015
Cited alongside, same era.
Prioritized experience replay
Original
Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David · 2015
Cited alongside, same era.
Multiagent cooperation and competition with deep reinforcement learning
Original
Tampuu, Ardi, Matiisen, Tambet, Kodelja, Dorian, Kuzovkin, Ilya, Korjus, Kristjan, Aru, Juhan, Aru, Jaan, and Vicente, Raul · 2015
Cited alongside, same era.

Then

A multi-agent framework for packet routing in wireless sensor networks
Ye, Dayong, Zhang, Minjie, and Yang, Yun · 2015
Later among the works it cites.
Learning to communicate with deep multi-agent reinforcement learning
Foerster, Jakob, Assael, Yannis M, de Freitas, Nando, and Whiteson, Shimon · 2016
Later among the works it cites.
Half field offense: an environment for multiagent learning and ad hoc teamwork
Hausknecht, Matthew, Mupparaju, Prannoy, Subramanian, Sandeep, Kalyanakrishnan, S, and Stone, P · 2016
Later among the works it cites.
Opponent modeling in deep reinforcement learning
He, He, Boyd-Graber, Jordan, Kwok, Kevin, and Daumé III, Hal · 2016
Later among the works it cites.
Learning to play guess who? and inventing a grounded language as a consequence
Original
Jorge, Emilio, Kågebäck, Mikael, and Gustavsson, Emil · 2016
Later among the works it cites.
Learning multiagent communication with backpropagation
Sukhbaatar, Sainbayar, Fergus, Rob, et al · 2016
Later among the works it cites.
Torchcraft: a library for machine learning research on real-time strategy games
Original
Synnaeve, Gabriel, Nardelli, Nantas, Auvolat, Alex, Chintala, Soumith, Lacroix, Timothée, Lin, Zeming, Richoux, Florian, and Usunier, Nicolas · 2016
Later among the works it cites.
Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks
Original
Usunier, Nicolas, Synnaeve, Gabriel, Lin, Zeming, and Chintala, Soumith · 2016
Later among the works it cites.
Coordinated deep reinforcement learners for traffic light control
Van der Pol, Elise and Oliehoek, Frans A · 2016
Later among the works it cites.
Sample efficient actor-critic with experience replay
Original
Wang, Ziyu, Bapst, Victor, Heess, Nicolas, Mnih, Volodymyr, Munos, Remi, Kavukcuoglu, Koray, and de Freitas, Nando · 2016
Later among the works it cites.
Offer: Off-environment reinforcement learning
Ciosek, Kamil and Whiteson, Shimon · 2017
Closest in time.
Multi-agent reinforcement learning in sequential social dilemmas
Original
Leibo, Joel Z, Zambaldi, Vinicius, Lanctot, Marc, Marecki, Janusz, and Graepel, Thore · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…