On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games

Understand

Softmax policy gradient is a popular algorithm for policy optimization in single-agent reinforcement learning, particularly since projection is not needed for each gradient update.

• However, in multi-agent systems, the lack of central coordination introduces significant additional difficulties in the convergence analysis.
• Even for a stochastic game with identical interest, there can be multiple Nash Equilibria (NEs), which disables proof techniques that rely on the existence of a unique global optimum.
• Moreover, the softmax parameterization introduces non-NE policies with zero gradient, making it difficult for gradient-based algorithms in seeking NEs.