Understanding Value Decomposition Algorithms in Deep Cooperative Multi-Agent Reinforcement Learning

Understand

Value function decomposition is becoming a popular rule of thumb for scaling up multi-agent reinforcement learning (MARL) in cooperative games.

• For such a decomposition rule to hold, the assumption of the individual-global max (IGM) principle must be made; that is, the local maxima on the decomposed value function per every agent must amount to the global maximum on the joint value function.
• This principle, however, does not have to hold in general.
• As a result, the applicability of value decomposition algorithms is concealed and their corresponding convergence properties remain unknown.