Independent and Decentralized Learning in Markov Potential Games

Understand

We study a multi-agent reinforcement learning dynamics, and analyze its asymptotic behavior in infinite-horizon discounted Markov potential games.

• We focus on the independent and decentralized setting, where players do not know the game parameters, and cannot communicate or coordinate.
• In each stage, players update their estimate of Q-function that evaluates their total contingent payoff based on the realized one-stage reward in an asynchronous manner.
• Then, players independently update their policies by incorporating an optimal one-stage deviation strategy based on the estimated Q-function.