2016

Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

Foerster, Jakob N., Assael, Yannis M., de Freitas, Nando et al.

Understand

We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks.

In these tasks, the agents are not given any pre-designed communication protocol.
Therefore, in order to successfully communicate, they must first automatically develop and agree upon their own communication protocol.
We present empirical results on two multi-agent learning problems based on well-known riddles, demonstrating that DDRQN can successfully solve such tasks and discover elegant communication protocols to do so.

Built on

Reinforcement learning for robots using neural networks
Lin, L.J · 1993
Earlier work this paper cites.
Multi-agent reinforcement learning: Independent vs. cooperative agents
Tan, M · 1993
Earlier work this paper cites.
Markov games as a framework for multi-agent reinforcement learning
Littman, M. L · 1994
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Reinforcement learning in the multi-robot domain
Matarić, M.J · 1997
Earlier work this paper cites.
Introduction to reinforcement learning
Sutton, R. S. and Barto, A. G · 1998
Earlier work this paper cites.
Probabilistic approach to collaborative multi-robot localization
Fox, D., Burgard, W., Kruppa, H., and Thrun, S · 2000
Earlier work this paper cites.
An algorithm for distributed reinforcement learning in cooperative multi-agent systems
Lauer, M. and Riedmiller, M · 2000
Earlier work this paper cites.
Learning communication for multi-agent systems
Giles, C. L. and Jim, K. C · 2002
Earlier work this paper cites.
100 prisoners and a lightbulb
Wu, W · 2002
Earlier work this paper cites.
A formal analysis and taxonomy of task allocation in multi-robot systems
Gerkey, B.P. and Matarić, M.J · 2004
Earlier work this paper cites.
Cooperative multi-agent learning: The state of the art
Panait, L. and Luke, S · 2005
Earlier work this paper cites.
Decentralized planning under uncertainty for teams of communicating agents
Spaan, M., Gordon, G. J., and Vlassis, N · 2006
Earlier work this paper cites.
Consensus and cooperation in networked multi-agent systems
Olfati-Saber, R., Fax, J.A., and Murray, R.M · 2007
Earlier work this paper cites.
If multi-agent learning is the answer, what is the question?
Shoham, Y., Powers, R., and Grenager, T · 2007
Earlier work this paper cites.

Similar

Learning of communication codes in multi-agent reinforcement learning problem
Kasai, T., Tenmoto, H., and Kamiya, A · 2008
Cited alongside, same era.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Cited alongside, same era.
Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations
Shoham, Y. and Leyton-Brown, K · 2009
Cited alongside, same era.
QueryPOMDP: POMDP-based communication in multiagent systems
Melo, F. S., Spaan, M., and Witwicki, S. J · 2011
Cited alongside, same era.
Are You Smart Enough to Work at Google?: Fiendish Puzzles and Impossible Interview Questions from the World’s Top Companies
Poundstone, W · 2012
Cited alongside, same era.
100 prisoners and a light bulb
Song, Y · 2012
Cited alongside, same era.
An overview of recent progress in the study of distributed multi-agent coordination
Cao, Y., Yu, W., Ren, W., and Chen, G · 2013
Cited alongside, same era.
Coordination of communication in robot teams by reinforcement learning
Maravall, D., De Lope, J., and Domínguez, R · 2013
Cited alongside, same era.
Coordinating multi-agent reinforcement learning with limited communication
Zhang, C. and Lesser, V · 2013
Cited alongside, same era.
Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning
Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang, X · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, D. and Ba, J · 2014
Cited alongside, same era.
Empirically evaluating multiagent learning algorithms
Zawadzki, E., Lipson, A., and Leyton-Brown, K · 2014
Cited alongside, same era.

Then

End-to-end training of deep visuomotor policies
Original
Levine, S., Finn, C., Darrell, T., and Abbeel, P · 2015
Later among the works it cites.
Recurrent reinforcement learning: A hybrid approach
Li, X., Li, L., Gao, J., He, X., Chen, J., Deng, L., and He, J · 2015
Later among the works it cites.
Move Evaluation in Go Using Deep Convolutional Neural Networks
Maddison, C. J., Huang, A., Sutskever, I., and Silver, D · 2015
Later among the works it cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Later among the works it cites.
Massively parallel methods for deep reinforcement learning
Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. De, Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., and Silver, D · 2015
Later among the works it cites.
Language understanding for text-based games using deep reinforcement learning
Narasimhan, K., Kulkarni, T., and Barzilay, R · 2015
Later among the works it cites.
Action-conditional video prediction using deep networks in Atari games
Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S · 2015
Later among the works it cites.
Incentivizing exploration in reinforcement learning with deep predictive models
Original
Stadie, B. C., Levine, S., and Abbeel, P · 2015
Later among the works it cites.
Multiagent cooperation and competition with deep reinforcement learning
Original
Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R · 2015
Later among the works it cites.
Dueling network architectures for deep reinforcement learning
Wang, Z., de Freitas, N., and Lanctot, M · 2015
Later among the works it cites.
Embed to control: A locally linear latent dynamics model for control from raw images
Watter, M., Springenberg, J. T., Boedecker, J., and Riedmiller, M. A · 2015
Later among the works it cites.
Increasing the action gap: New operators for reinforcement learning
Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., and Munos, R · 2016
Closest in time.
Prioritized experience replay
Schaul, T., Quan, J., Antonoglou, I., and Silver, D · 2016
Closest in time.
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D · 2016
Closest in time.
Deep reinforcement learning with double Q-learning
van Hasselt, H., Guez, A., and Silver, D · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…