2020

ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

Wang, Tonghan, Dong, Heng, Lesser, Victor et al.

Understand

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors.

However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors.
In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks.
In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA).

Built on

The wealth of nations [1776], 1937
Smith, A · 1937
Earlier work this paper cites.
Multi-agent reinforcement learning: Independent vs. cooperative agents
Tan, M · 1993
Earlier work this paper cites.
The organization of work in social insect colonies
Gordon, D. M · 1996
Earlier work this paper cites.
Rope: Role oriented programming environment for multiagent systems
Becht, M., Gurzki, T., Klarmann, J., and Muscholl, M · 1999
Earlier work this paper cites.
Task decomposition, dynamic role assignment, and low-bandwidth communication for real-time strategic teamwork
Stone, P. and Veloso, M · 1999
Earlier work this paper cites.
Soda: Societies and infrastructures in the analysis and design of agent-based systems
Omicini, A · 2000
Earlier work this paper cites.
The gaia methodology for agent-oriented analysis and design
Wooldridge, M., Jennings, N. R., and Kinny, D · 2000
Earlier work this paper cites.
Roles in agent-oriented modeling
Depke, R., Heckel, R., and Küster, J. M · 2001
Earlier work this paper cites.
Multiagent planning with factored mdps
Guestrin, C., Koller, D., and Parr, R · 2002
Earlier work this paper cites.
Prometheus: A methodology for developing intelligent agents
Padgham, L. and Winikoff, M · 2002
Earlier work this paper cites.
From agents to organizations: an organizational view of multi-agent systems
Ferber, J., Gutknecht, O., and Michel, F · 2003
Earlier work this paper cites.
Agent oriented software engineering with ingenias
Pavón, J. and Gómez-Sanz, J · 2003
Earlier work this paper cites.
Tropos: An agent-oriented software development methodology
Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., and Mylopoulos, J · 2004
Earlier work this paper cites.
A metamodel for agents, roles, and groups
Odell, J., Nodine, M., and Levy, R · 2004
Earlier work this paper cites.
The passi and agile passi mas meta-models compared with a unifying proposal
Cossentino, M., Gaglio, S., Sabatucci, L., and Seidita, V · 2005
Earlier work this paper cites.
Emergence of division of labour in halictine bees: contributions of social interactions and behavioural variance
Jeanson, R., Kukuk, P. F., and Fewell, J. H · 2005
Earlier work this paper cites.
Graphical models, exponential families, and variational inference
Wainwright, M. J., Jordan, M. I., et al · 2008
Earlier work this paper cites.
Role-based multi-agent systems
Zhu, H. and Zhou, M · 2008
Earlier work this paper cites.
O-mase: a customisable approach to designing and building complex, adaptive multi-agent systems
DeLoach, S. A. and Garcia-Ojeda, J. C · 2010
Earlier work this paper cites.
Using aseme methodology for model-driven agent systems development
Spanoudakis, N. and Moraitis, P · 2010
Earlier work this paper cites.
Coordinated multi-agent reinforcement learning in networked distributed pomdps
Zhang, C. and Lesser, V · 2011
Earlier work this paper cites.

Similar

The condensed wealth of nations
Butler, E · 2012
Cited alongside, same era.
An overview of recent progress in the study of distributed multi-agent coordination
Cao, Y., Yu, W., Ren, W., and Chen, G · 2012
Cited alongside, same era.
Game theory and multi-agent reinforcement learning
Nowé, A., Vrancx, P., and De Hauwere, Y.-M · 2012
Cited alongside, same era.
Adelfe 2.0: Handbook on agent-oriented design processes, m. cossentino, v. hilaire, a. molesini, and v. seidita, 2014
Bonjean, N., Mefteh, W., Gleizes, M., Maurel, C., and Migeon, F · 2014
Cited alongside, same era.
Learning phrase representations using rnn encoder–decoder for statistical machine translation
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y · 2014
Cited alongside, same era.
Learning to communicate with deep multi-agent reinforcement learning
Foerster, J., Assael, I. A., de Freitas, N., and Whiteson, S · 2016
Cited alongside, same era.
A concise introduction to decentralized POMDPs , volume 1
Oliehoek, F. A., Amato, C., et al · 2016
Cited alongside, same era.
Learning multiagent communication with backpropagation
Sukhbaatar, S., Fergus, R., et al · 2016
Cited alongside, same era.
Deep variational information bottleneck
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K · 2017
Cited alongside, same era.
Learning cooperative visual dialog agents with deep reinforcement learning
Das, A., Kottur, S., Moura, J. M., Lee, S., and Batra, D · 2017
Cited alongside, same era.
Stabilising experience replay for deep multi-agent reinforcement learning
Foerster, J., Nardelli, N., Farquhar, G., Afouras, T., Torr, P. H., Kohli, P., and Whiteson, S · 2017
Cited alongside, same era.
Vain: Attentional multi-agent predictive modeling
Hoshen, Y · 2017
Cited alongside, same era.

Then

Learning attentional communication for multi-agent cooperation
Jiang, J. and Lu, Z · 2018
Later among the works it cites.
Neural relational inference for interacting systems
Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., and Zemel, R · 2018
Later among the works it cites.
Role-based modeling for designing agent behavior in self-organizing multi-agent systems
Lhaksmana, K. M., Murakami, Y., and Ishida, T · 2018
Later among the works it cites.
Emergence of grounded compositional language in multi-agent populations
Mordatch, I. and Abbeel, P · 2018
Later among the works it cites.
Credit assignment for collective multiagent rl with global rewards
Nguyen, D. T., Kumar, A., and Lau, H. C · 2018
Later among the works it cites.
Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning
Rashid, T., Samvelyan, M., Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S · 2018
Later among the works it cites.
Value-decomposition networks for cooperative multi-agent learning based on team reward
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al · 2018
Later among the works it cites.
Glomo: unsupervised learning of transferable relational graphs
Yang, Z., Zhao, J., Dhingra, B., He, K., Cohen, W. W., Salakhutdinov, R. R., and LeCun, Y · 2018
Later among the works it cites.
Tarmac: Targeted multi-agent communication
Das, A., Gervet, T., Romoff, J., Batra, D., Parikh, D., Rabbat, M., and Pineau, J · 2019
Later among the works it cites.
Actor-attention-critic for multi-agent reinforcement learning
Iqbal, S. and Sha, F · 2019
Later among the works it cites.
Learning fairness in multi-agent systems
Jiang, J. and Lu, Z · 2019
Later among the works it cites.
Learning to schedule communication in multi-agent reinforcement learning
Kim, D., Moon, S., Hostallero, D., Kang, W. J., Lee, T., Son, K., and Yi, Y · 2019
Later among the works it cites.
Maven: Multi-agent variational exploration
Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S · 2019
Later among the works it cites.
The starcraft multi-agent challenge
Original
Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S · 2019
Later among the works it cites.
Learning when to communicate at scale in multiagent cooperative and competitive tasks
Singh, A., Jain, T., and Sukhbaatar, S · 2019
Later among the works it cites.
Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning
Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y · 2019
Later among the works it cites.
Grandmaster level in starcraft ii using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al · 2019
Later among the works it cites.
Probabilistic recursive reasoning for multi-agent reinforcement learning
Wen, Y., Yang, Y., Luo, R., Wang, J., and Pan, W · 2019
Later among the works it cites.
Emergent tool use from multi-agent autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., and Mordatch, I · 2020
Closest in time.
Incorporating pragmatic reasoning communication into emergent language
Original
Kang, Y., Wang, T., and de Melo, G · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…