Neural Replicator Dynamics

Understand

Policy gradient and actor-critic algorithms form the basis of many commonly used training techniques in deep reinforcement learning.

• Using these algorithms in multiagent environments poses problems such as nonstationarity and instability.
• In this paper, we first demonstrate that standard softmax-based policy gradient can be prone to poor performance in the presence of even the most benign nonstationarity.
• By contrast, it is known that the replicator dynamics, a well-studied model from evolutionary game theory, eliminates dominated strategies and exhibits convergence of the time-averaged trajectories to interior Nash equilibria in zero-sum games.