Model-Free Opponent Shaping

Understand

In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD).

• To overcome this, some methods, such as Learning with Opponent-Learning Awareness (LOLA), shape their opponents' learning process.
• However, these methods are myopic since only a small number of steps can be anticipated, are asymmetric since they treat other agents as naive learners, and require the use of higher-order derivatives, which are calculated through white-box access to an opponent's differentiable learning algorithm.
• To address these issues, we propose Model-Free Opponent Shaping (M-FOS).