COLA: Consistent Learning with Opponent-Learning Awareness

Understand

Learning in general-sum games is unstable and frequently leads to socially undesirable (Pareto-dominated) outcomes.

• To mitigate this, Learning with Opponent-Learning Awareness (LOLA) introduced opponent shaping to this setting, by accounting for each agent's influence on their opponents' anticipated learning steps.
• However, the original LOLA formulation (and follow-up work) is inconsistent because LOLA models other agents as naive learners rather than LOLA agents.
• In previous work, this inconsistency was suggested as a cause of LOLA's failure to preserve stable fixed points (SFPs).