Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

Understand

Many advances in cooperative multi-agent reinforcement learning (MARL) are based on two common design principles: value decomposition and parameter sharing.

• A typical MARL algorithm of this fashion decomposes a centralized Q-function into local Q-networks with parameters shared across agents.
• Such an algorithmic paradigm enables centralized training and decentralized execution (CTDE) and leads to efficient learning in practice.
• Despite all the advantages, we revisit these two principles and show that in certain scenarios, e.g., environments with a highly multi-modal reward landscape, value decomposition, and parameter sharing can be problematic and lead to undesired outcomes.