2018

Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications

Nguyen, Thanh Thi, Nguyen, Ngoc Duy, Nahavandi, Saeid

Understand

Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision-making problems.

These algorithms however have faced great challenges when dealing with high-dimensional environments.
The recent development of deep learning has enabled RL methods to drive optimal policies for sophisticated and capable agents, which can perform efficiently in these challenging environments.
This paper addresses an important aspect of deep RL related to situations that require multiple agents to communicate and cooperate to solve complex tasks.

Built on

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., … and Kavukcuoglu, K. (2016, June). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (pp. 1928-1937)
1937
Earlier work this paper cites.
Bellman, R. (1952). On the theory of dynamic programming. Proceedings of the National Academy of Sciences , 38(8), 716-719
1952
Earlier work this paper cites.
Minsky, M. L. (1954). Theory of neural-analog reinforcement systems and its application to the brain model problem. Princeton University
1954
Earlier work this paper cites.
Klopf, A. (1972). Brain function and adaptive systems: a heterostatic theory. Air Force Cambridge Research Laboratories Special Report
1972
Earlier work this paper cites.
Mahadevan, S., and Connell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence , 55(2-3), 311-365
1992
Earlier work this paper cites.
Watkins, C. J., and Dayan, P. (1992). Q-learning. Machine Learning , 8(3-4), 279-292
1992
Earlier work this paper cites.
Tan, M. (1993). Multi-agent reinforcement learning: independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning (pp. 330-337)
1993
Earlier work this paper cites.
Benbrahim, H., and Franklin, J. A. (1997). Biped dynamic walking using reinforcement learning. Robotics and Autonomous Systems , 22(3-4), 283-302
1997
Earlier work this paper cites.
Schaal, S. (1997). Learning from demonstration. In Advances in Neural Information Processing Systems (pp. 1040-1046)
1997
Earlier work this paper cites.
Tsitsiklis, J. N., and Van Roy, B. (1997). Analysis of temporal-diffference learning with function approximation. In Advances in Neural Information Processing Systems (pp. 1075-1081)
1997
Earlier work this paper cites.
Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction. MIT press
1998
Earlier work this paper cites.
Konda, V. R., and Tsitsiklis, J. N. (2000). Actor-critic algorithms. In Advances in Neural Information Processing Systems (pp. 1008-1014)
2000
Earlier work this paper cites.
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016, June). Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning (pp. 1995-2003)
2003
Earlier work this paper cites.
de Cote, E. M., Lazaric, A., and Restelli, M. (2006, May). Learning to cooperate in multi-agent social dilemmas. In Proceedings of the 5th International Joint Conference on Autonomous Agents and Multiagent Systems (pp. 783-785). ACM
2006
Earlier work this paper cites.
Zinkevich, M., Greenwald, A., and Littman, M. L. (2006). Cyclic equilibria in Markov games. In Advances in Neural Information Processing Systems (pp. 1641-1648)
2006
Earlier work this paper cites.
Harati, A., Ahmadabadi, M. N., and Araabi, B. N. (2007). Knowledge-based multiagent credit assignment: a study on task type and critic information. IEEE Systems Journal , 1(1), 55-67
2007
Earlier work this paper cites.
Matignon, L., Laurent, G., and Le Fort-Piat, N. (2007, October). Hysteretic Q-Learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 64-69)
2007
Earlier work this paper cites.
Shoham, Y., Powers, R., and Grenager, T. (2007). If multi-agent learning is the answer, what is the question?. Artificial Intelligence , 171(7), 365-377
2007
Earlier work this paper cites.
Busoniu, L., Babuska, R., and De Schutter, B. (2008). A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews , 38 (2), 156-172
2008
Earlier work this paper cites.
Riedmiller, M., Gabel, T., Hafner, R., and Lange, S. (2009). Reinforcement learning for robot soccer. Autonomous Robots , 27(1), 55-73
2009
Earlier work this paper cites.
Hasselt, H. V. (2010). Double Q-learning. In Advances in Neural Information Processing Systems (pp. 2613-2621)
2010
Earlier work this paper cites.
Janssen, M. A., Holahan, R., Lee, A., and Ostrom, E. (2010). Lab experiments for the study of social-ecological systems. Science , 328(5978), 613-617
2010
Earlier work this paper cites.
Brandouy, O., Mathieu, P., and Veryzhenko, I. (2011, January). On the design of agent-based artificial stock markets. In International Conference on Agents and Artificial Intelligence (pp. 350-364)
2011
Earlier work this paper cites.
Chung, T. H., Hollinger, G. A., and Isler, V. (2011). Search and pursuit-evasion in mobile robotics. Autonomous Robots , 31(4), 299
2011
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (pp. 1097-1105)
2012
Earlier work this paper cites.
Oliehoek, F. A. (2012). Decentralized POMDPs. In Reinforcement Learning (pp. 471-503). Springer, Berlin, Heidelberg
2012
Earlier work this paper cites.
Abdallah, S., and Kaisers, M. (2013, May). Addressing the policy-bias of Q-learning by repeating updates. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (pp. 1045-1052)
2013
Earlier work this paper cites.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learning environment: an evaluation platform for general agents. Journal of Artificial Intelligence Research , 47, 253-279
2013
Earlier work this paper cites.
Mulling, K., Kober, J., Kroemer, O., and Peters, J. (2013). Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research , 32(3), 263-279
2013
Earlier work this paper cites.
Deng, L., and Yu, D. (2014). Deep learning: methods and applications. Foundations and Trends in Signal Processing , 7(3–4), 197-387
2014
Earlier work this paper cites.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January). Deterministic policy gradient algorithms. In International Conference on Machine Learning (pp. 387-395)
2014
Earlier work this paper cites.
Bloembergen, D., Tuyls, K., Hennes, D., and Kaisers, M. (2015). Evolutionary dynamics of multi-agent learning: a survey. Journal of Artificial Intelligence Research , 53, 659-697
2015
Earlier work this paper cites.
Fernandez-Gauna, B., Etxeberria-Agiriano, I., and Grana, M. (2015). Learning multirobot hose transportation and deployment by distributed round-robin Q-learning. PloS One , 10(7), e0127129
2015
Earlier work this paper cites.
Hausknecht, M., and Stone, P. (2015). Deep recurrent Q-learning for partially observable MDPs. CoRR, abs/1507.06527 , 7(1)
Original
2015
Earlier work this paper cites.
He, J., Peng, J., Jiang, F., Qin, G., and Liu, W. (2015). A distributed Q learning spectrum decision scheme for cognitive radio sensor network. International Journal of Distributed Sensor Networks , 11(5), 301317
2015
Earlier work this paper cites.
Heess, N., Hunt, J. J., Lillicrap, T. P., and Silver, D. (2015). Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455
Original
2015
Earlier work this paper cites.
Hussin, M., Hamid, N. A. W. A., and Kasmiran, K. A. (2015). Improving reliability in resource management through adaptive reinforcement learning for distributed systems. Journal of Parallel and Distributed Computing , 75, 93-100
2015
Earlier work this paper cites.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., … and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971
Original
2015
Earlier work this paper cites.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., … and Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature , 518(7540), 529-533
2015
Earlier work this paper cites.
Parisotto, E., Ba, J. L., and Salakhutdinov, R. (2015). Actor-mimic: deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342
Original
2015
Earlier work this paper cites.
Rahman, M. S., Mahmud, M. A., Pota, H. R., Hossain, M. J., and Orchi, T. F. (2015). Distributed multi-agent-based protection scheme for transient stability enhancement in power systems. International Journal of Emerging Electric Power Systems , 16(2), 117-129
2015
Earlier work this paper cites.

Similar

Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., … and Hadsell, R. (2015). Policy distillation. arXiv preprint arXiv:1511.06295
Original
2015
Cited alongside, same era.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952
Original
2015
Cited alongside, same era.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, June). Trust region policy optimization. In International Conference on Machine Learning (pp. 1889-1897)
2015
Cited alongside, same era.
Sorokin, I., Seleznev, A., Pavlov, M., Fedorov, A., and Ignateva, A. (2015). Deep attention recurrent Q-network. arXiv preprint arXiv:1512.01693
Original
2015
Cited alongside, same era.
Yu, C., Zhang, M., Ren, F., and Tan, G. (2015). Multiagent learning of coordination in loosely coupled multiagent systems. IEEE Transactions on Cybernetics , 45(12), 2853-2867
2015
Cited alongside, same era.
Yu, C., Zhang, M., Ren, F., and Tan, G. (2015). Emotional multiagent reinforcement learning in spatial social dilemmas. IEEE Transactions on Neural Networks and Learning Systems , 26(12), 3083-3096
2015
Cited alongside, same era.
Abdallah, S., and Kaisers, M. (2016). Addressing environment non-stationarity by repeating Q-learning updates. The Journal of Machine Learning Research , 17(1), 1582-1612
2016
Cited alongside, same era.
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., … and Schrittwieser, J. (2016). DeepMind lab. arXiv preprint arXiv:1612.03801
Original
2016
Cited alongside, same era.
Castaneda, A. O. (2016). Deep Reinforcement Learning Variants of Multi-Agent Learning Algorithms (Master’s Thesis, School of Informatics, University of Edinburgh)
2016
Cited alongside, same era.
Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016, June). Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning (pp. 1329-1338)
2016
Cited alongside, same era.
Egorov, M. (2016). Multi-agent deep reinforcement learning. Stanford University
2016
Cited alongside, same era.
Foerster, J., Assael, Y. M., de Freitas, N., and Whiteson, S. (2016). Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (pp. 2137-2145)
2016
Cited alongside, same era.

Then

Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Perolat, J., Silver, D., and Graepel, T. (2017). A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems (pp. 4193-4206)
2017
Later among the works it cites.
Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017, May). Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (pp. 464-473)
2017
Later among the works it cites.
Lerer, A., and Peysakhovich, A. (2017). Maintaining cooperation in complex social dilemmas using deep reinforcement learning. arXiv preprint arXiv:1707.01068
Original
2017
Later among the works it cites.
Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274
Original
2017
Later among the works it cites.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O. P., and Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems (pp. 6379-6390)
2017
Later among the works it cites.
Mousavi, S. S., Schukat, M., and Howley, E. (2017). Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intelligent Transport Systems , 11(7), 417-423
2017
Later among the works it cites.
Nahavandi, S. (2017). Trusted autonomy between humans and robots: toward human-on-the-loop in robotics and autonomous systems. IEEE Systems, Man, and Cybernetics Magazine , 3(1), 10-17
2017
Later among the works it cites.
Nguyen, N. D., Nguyen, T., and Nahavandi, S. (2017). System design perspective for human-level agents using deep reinforcement learning: a survey. IEEE Access , 5, 27091-27102
2017
Later among the works it cites.
Noureddine, D., Gharbi, A. and Ahmed, S. (2017). Multi-agent deep reinforcement learning for task allocation in dynamic environment. In Proceedings of the 12th International Conference on Software Technologies (ICSOFT) , pp. 17-26
2017
Later among the works it cites.
Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J. (2017, July). Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In International Conference on Machine Learning (pp. 2681-2690)
2017
Later among the works it cites.
Perolat, J., Leibo, J. Z., Zambaldi, V., Beattie, C., Tuyls, K., and Graepel, T. (2017). A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems (pp. 3643-3652)
2017
Later among the works it cites.
Piot, B., Geist, M., and Pietquin, O. (2017). Bridging the gap between imitation learning and inverse reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 28(8), 1814-1826
2017
Later among the works it cites.
Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., and Vicente, R. (2017). Multiagent cooperation and competition with deep reinforcement learning. PloS One , 12(4), e0172395
2017
Later among the works it cites.
Yin, H., and Pan, S. J. (2017, January). Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (pp. 1640-1646)
2017
Later among the works it cites.
Zhu, P., Li, X., and Poupart, P. (2017). On improving deep reinforcement learning for POMDPs. arXiv preprint arXiv:1704.07978
Original
2017
Later among the works it cites.
Calvo, J. J. A., and Dusparic, I. (2018). Heterogeneous multi-agent deep reinforcement learning for traffic lights control. The 26th Irish Conference on Artificial Intelligence and Cognitive Science (pp. 1-12), Dublin, Ireland
2018
Closest in time.
Corneil, D., Gerstner, W., and Brea, J. (2018). Efficient model-based deep reinforcement learning with variational state tabulation. arXiv preprint arXiv:1802.04325
Original
2018
Closest in time.
Foerster, J. N., Song, F., Hughes, E., Burch, N., Dunning, I., Whiteson, S., … and Bowling, M. (2018). Bayesian action decoder for deep multi-agent reinforcement learning. arXiv preprint arXiv:1811.01458
Original
2018
Closest in time.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018). Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (pp. 2974-2982)
2018
Closest in time.
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2018, July). Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (pp. 122-130)
2018
Closest in time.
Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2018). Is multiagent deep reinforcement learning the answer or the question? a brief survey. arXiv preprint arXiv:1810.05587
Original
2018
Closest in time.
Hong, Z. W., Su, S. Y., Shann, T. Y., Chang, Y. H., and Lee, C. Y. (2018, July). A deep policy inference Q-network for multi-agent systems. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (pp. 1388-1396)
2018
Closest in time.
Kattepur, A., Rath, H. K., Simha, A., and Mukherjee, A. (2018, April). Distributed optimization in multi-agent robotics for industry 4.0 warehouses. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing (pp. 808-815). ACM
2018
Closest in time.
Kilinc, O., and Montana, G. (2018). Multi-agent deep reinforcement learning with extremely noisy observations. arXiv preprint arXiv:1812.00922
Original
2018
Closest in time.
Lin, K., Zhao, R., Xu, Z., and Zhou, J. (2018). Efficient large-scale fleet management via multi-agent deep reinforcement learning. arXiv preprint arXiv:1802.06444
Original
2018
Closest in time.
Luviano-Cruz, D., Garcia-Luna, F., Perez-Dominguez, L., and Gadi, S. (2018). Multi-agent reinforcement learning using linear fuzzy model applied to cooperative mobile robots. Symmetry , 10(10), 461
2018
Closest in time.
Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. (2018, May). Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7559-7566). IEEE
2018
Closest in time.
Nguyen, T. (2018). A multi-objective deep reinforcement learning framework. arXiv preprint arXiv:1803.02965
Original
2018
Closest in time.
Nguyen, N. D., Nahavandi, S., and Nguyen, T. (2018). A human mixed strategy approach to deep reinforcement learning. In 2018 IEEE International Conference on Systems, Man, and Cybernetics , (pp. 4023-4028)
2018
Closest in time.
Nguyen, T., Nguyen, N. D., and Nahavandi, S. (2018). Multi-agent deep reinforcement learning with human strategies. arXiv preprint arXiv:1806.04562
Original
2018
Closest in time.
Palmer, G., Tuyls, K., Bloembergen, D., and Savani, R. (2018, July). Lenient multi-agent deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (pp. 443-451)
2018
Closest in time.
Pendharkar, P. C., and Cusatis, P. (2018). Trading financial indices with reinforcement learning agents. Expert Systems with Applications , 103, 1-13
2018
Closest in time.
Prasad, A., and Dusparic, I. (2018). Multi-agent deep reinforcement learning for zero energy communities. arXiv preprint arXiv:1810.03679
Original
2018
Closest in time.
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. (2018). QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv:1803.11485
Original
2018
Closest in time.
Schmid, K., Belzner, L., Gabor, T., and Phan, T. (2018, October). Action markets in deep multi-agent reinforcement learning. In International Conference on Artificial Neural Networks (pp. 240-249)
2018
Closest in time.
Serban, I. V., Sankar, C., Pieper, M., Pineau, J., and Bengio, Y. (2018). The bottleneck simulator: a model-based deep reinforcement learning approach. arXiv preprint arXiv:1807.04723
Original
2018
Closest in time.
Silva, F.L., Taylor, M.E., and Costa, A.H.R. (2018). Autonomously reusing knowledge in multiagent reinforcement learning. In The 27th International Joint Conference on Artificial Intelligence , pp. 5487-5493
2018
Closest in time.
Zheng, Y., Meng, Z., Hao, J., and Zhang, Z. (2018, August). Weighted double deep multiagent reinforcement learning in stochastic cooperative environments. In Pacific Rim International Conference on Artificial Intelligence (pp. 421-429)
2018
Closest in time.
Schwab, D., Zhu, Y., and Veloso, M. (2018, July). Zero shot transfer learning for robot soccer. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (pp. 2070-2072)
2072
Closest in time.
Hasselt, H. V., Guez, A., and Silver, D. (2016, February). Deep reinforcement learning with double Q-Learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (pp. 2094-2100). AAAI Press
2094
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…