QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning

Understand

In Cooperative Multi-Agent Reinforcement Learning (MARL) and under the setting of Centralized Training with Decentralized Execution (CTDE), agents observe and interact with their environment locally and independently.

• With local observation and random sampling, the randomness in rewards and observations leads to randomness in long-term returns.
• Existing methods such as Value Decomposition Network (VDN) and QMIX estimate the value of long-term returns as a scalar that does not contain the information of randomness.
• Our proposed model QR-MIX introduces quantile regression, modeling joint state-action values as a distribution, combining QMIX with Implicit Quantile Network (IQN).