Competitive Policy Optimization

Understand

A core challenge in policy optimization in competitive Markov decision processes is the design of efficient optimization methods with desirable convergence and stability properties.

• To tackle this, we propose competitive policy optimization (CoPO), a novel policy gradient approach that exploits the game-theoretic nature of competitive games to derive policy updates.
• Motivated by the competitive gradient optimization method, we derive a bilinear approximation of the game objective.
• In contrast, off-the-shelf policy gradient methods utilize only linear approximations, and hence do not capture interactions among the players.