Reevaluating Policy Gradient Methods for Imperfect-Information Games

Understand

In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR).

• In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches.
• To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for five large games.
• Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games.