2019

Learning in Discounted-cost and Average-cost Mean-field Games

Anahtarcı, Berkay, Karıksız, Can Deha, Saldi, Naci

Understand

We consider learning approximate Nash equilibria for discrete-time mean-field games with nonlinear stochastic state dynamics subject to both average and discounted costs.

To this end, we introduce a mean-field equilibrium (MFE) operator, whose fixed point is a mean-field equilibrium (i.e.
equilibrium in the infinite population limit).
We first prove that this operator is a contraction, and propose a learning algorithm to compute an approximate mean-field equilibrium by approximating the MFE operator with a random one.

Built on

Linear-quadratic mean-field reinforcement learning: convergence of policy gradient methods
Original
R. Carmona, M. Lauriere, and Z. Tan · 1910
Earlier work this paper cites.
Model-free mean-field reinforcement learning: Mean-field MDP and mean-field Q-learning
Original
R. Carmona, M. Lauriere, and Z. Tan · 1910
Earlier work this paper cites.
Convergence of dynamic programming models
H.J. Langen · 1981
Earlier work this paper cites.
Convergence of Stochastic Processes
D. Pollard · 1984
Earlier work this paper cites.
Adaptive Markov Control Processes
O. Hernández-Lerma · 1989
Earlier work this paper cites.
Recurrence conditions for Markov decision processes with Borel state space: a survey
O. Hernández-Lerma, R. Montes-De-Oca, and R. Cavazos-Cadena · 1991
Earlier work this paper cites.
Discrete-Time Markov Control Processes: Basic Optimality Criteria
O. Hernández-Lerma and J.B. Lasserre · 1996
Earlier work this paper cites.
Further Topics on Discrete-Time Markov Control Processes
O. Hernández-Lerma and J.B. Lasserre · 1999
Earlier work this paper cites.
Perturbation Analysis of Optimization Problems
J.F. Bonnans and A. Shapiro · 2000
Earlier work this paper cites.
Q-learning in regularized mean-field games
Original
B. Anahtarci, C.D. Kariksiz, and N. Saldi · 2003
Earlier work this paper cites.
Oblivious equilibrium: A mean field approximation for large-scale dynamic games
Gabriel Weintraub, Lanier Benkard, and Benjamin Van Roy · 2005
Earlier work this paper cites.
Infinite Dimensional Analysis
C.D. Aliprantis and K.C. Border · 2006
Earlier work this paper cites.
Ergodic properties of Markov processes
M. Hairer · 2006
Earlier work this paper cites.
Large population stochastic dynamic games: Closed loop McKean-Vlasov systems and the Nash certainty equivalence principle
M. Huang, R.P. Malhamé, and P.E. Caines · 2006
Earlier work this paper cites.
Large-population cost coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized ϵ \epsilon -Nash equilibria
M. Huang, P.E. Caines, and R.P. Malhamé · 2007
Earlier work this paper cites.
Mean field games
J. Lasry and P.Lions · 2007
Earlier work this paper cites.
Concentration inequalities for dependent random variables via the martingale method
L. Kontorovich and K. Ramanan · 2008
Earlier work this paper cites.
Finite-time bounds for fitted value iteration
R. Munos and C. Szepesvári · 2008
Earlier work this paper cites.

Similar

Markov perfect industry dynamics with many firms
Gabriel Y. Weintraub, C. Lanier Benkard, and Benjamin Van Roy · 2008
Cited alongside, same era.
Discrete time, finite state space mean field games
D.A. Gomes, J. Mohr, and R.R. Souza · 2010
Cited alongside, same era.
Large-population LQG games involving major player: The Nash certainty equivalence principle
M. Huang · 2010
Cited alongside, same era.
Learning and Generalization: With Applications to Neural Networks
M. Vidyasagar · 2010
Cited alongside, same era.
Computational methods for oblivious equilibrium
Gabriel Y. Weintraub, C. Lanier Benkard, and Benjamin Van Roy · 2010
Cited alongside, same era.
Notes on Mean-field Games
P. Cardaliaguet · 2011
Cited alongside, same era.
Gibbs Measures and Phase Transitions
H.O. Georgii · 2011
Cited alongside, same era.
Multiagent learning in large anonymous games
Ian Kash, Eric Friedman, and Joseph Halpern · 2011
Cited alongside, same era.
Mean Field Games and Mean Field Type Control Theory
A. Bensoussan, J. Frehse, and P. Yam · 2013
Cited alongside, same era.
Probabilistic analysis of mean-field games
R. Carmona and F. Delarue · 2013
Cited alongside, same era.
Discrete time mean-field stochastic linear-quadratic optimal control problems
R. Elliot, X. Li, and Y. Ni · 2013
Cited alongside, same era.
Linear-quadratic-Gaussian mean field games under high rate quantization
M. Nourian and G.N. Nair · 2013
Cited alongside, same era.

Then

Discrete-time decentralized control using the risk-sensitive performance criterion in the large population regime: a mean field approach
J. Moon and T. Başar · 2015
Later among the works it cites.
Stationary anonymous sequential games with undiscounted rewards
P. Wiecek and E. Altman · 2015
Later among the works it cites.
Discrete-time mean field Stackelberg games with a large number of followers
J. Moon and T. Başar · 2016
Later among the works it cites.
Markov–Nash equilibria in mean-field games with discounted cost
N. Saldi, T. Başar, and M. Raginsky · 2018
Later among the works it cites.
A policy gradient algorithm to compute boundedly rational stationary mean field equilibria
Jayakumar Subramanian and Aditya Mahajan · 2018
Later among the works it cites.
Reinforcement learning: Theory and algorithms
A. Agarwal, N. Jiang, and S. Kakade · 2019
Closest in time.
Approximate fictitious play for mean-field games
Original
R. Elie, J. Perolat, M. Lauriere, M. Geist, and O. Pietquin · 2019
Closest in time.
Actor-critic provably finds Nash equilibria of linear-quadratic mean-field games
Original
Z. Fu, Z. Yang, Y. Chen, and Z. Wang · 2019
Closest in time.
Learning mean-field games
X. Guo, A. Hu, R. Xu, and J. Zhang · 2019
Closest in time.
Statistical learning theory
B. Hajek and M. Raginsky · 2019
Closest in time.
Binary mean field stochastic games: Stationary equilibria and comparative statics
Minyi Huang and Yan Ma · 2019
Closest in time.
Approximate Nash equilibria in partially observed stochastic games with mean-field interactions
N. Saldi, T. Başar, and M. Raginsky · 2019
Closest in time.
Reinforcement learning in stationary mean-field games
Jayakumar Subramanian and Aditya Mahajan · 2019
Closest in time.
Discrete-time ergodic mean-field games with average reward on compact spaces
P. Wiecek · 2019
Closest in time.
Discrete-time average-cost mean-field games on Polish spaces
N. Saldi · 2020
Closest in time.
Approximate Markov-Nash equilibria for discrete-time risk-sensitive mean-field games
N. Saldi, T. Başar, and M. Raginsky · 2020
Closest in time.
Mean field equilibrium: Uniqueness, existence, and comparative statics
Bar Light and Gabriel Y. Weintraub · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…