2019

Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

Carmona, René, Laurière, Mathieu, Tan, Zongjun

Understand

We study infinite horizon discounted Mean Field Control (MFC) problems with common noise through the lens of Mean Field Markov Decision Processes (MFMDP).

We allow the agents to use actions that are randomized not only at the individual level but also at the level of the population.
This common randomization allows us to establish connections between both closed-loop and open-loop policies for MFC and Markov policies for the MFMDP.
In particular, we show that there exists an optimal closed-loop policy for the original MFC.

Built on

Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games: Ii–the finite horizon case
Original
Carmona, R. and Laurière, M. (2019) · 1908
Earlier work this paper cites.
Numerical resolution of mckean-vlasov fbsdes using neural networks
Original
Germain, M., Mikael, J., and Warin, X. (2019) · 1909
Earlier work this paper cites.
Dynamic programming principles for mean-field controls with learning
Original
Gu, H., Guo, X., Wei, X., and Xu, R. (2019) · 1911
Earlier work this paper cites.
Applications of the deep galerkin method to solving partial integro-differential and hamilton-jacobi-bellman equations
Original
Al-Aradi, A., Correia, A., Naiff, D. d. F., Jardim, G., and Saporito, Y. (2019) · 1912
Earlier work this paper cites.
Mean-field markov decision processes with common noise and open-loop controls
Original
Motte, M. and Pham, H. (2019) · 1912
Earlier work this paper cites.
Mean-field controls with q-learning for cooperative marl: Convergence and complexity analysis
Original
Gu, H., Guo, X., Wei, X., and Xu, R. (2020) · 2002
Earlier work this paper cites.
Q-learning in regularized mean-field games
Original
Anahtarci, B., Kariksiz, C. D., and Saldi, N. (2020) · 2003
Earlier work this paper cites.
Learning rates for Q-learning
Even-Dar, E. and Mansour, Y. (2003) · 2003
Earlier work this paper cites.
Stochastic optimal control: the discrete-time case
Bertsekas, D. P. and Shreve, S. (2004) · 2004
Earlier work this paper cites.
Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle
Huang, M., Malhamé, R. P., and Caines, P. E. (2006) · 2006
Earlier work this paper cites.
Mean field games
Lasry, J.-M. and Lions, P.-L. (2007) · 2007
Earlier work this paper cites.
Mean field games: numerical methods
Achdou, Y. and Capuzzo-Dolcetta, I. (2010) · 2010
Earlier work this paper cites.

Similar

Action-gap phenomenon in reinforcement learning
Farahmand, A.-m. (2011) · 2011
Cited alongside, same era.
Mean field games: numerical methods for the planning problem
Achdou, Y., Camilli, F., and Capuzzo-Dolcetta, I. (2012) · 2012
Cited alongside, same era.
Mean field games and mean field type control theory
Bensoussan, A., Frehse, J., and Yam, S. C. P. (2013) · 2013
Cited alongside, same era.
A probabilistic approach to classical solutions of the master equation for large population equilibria
Original
Chassagneux, J.-F., Crisan, D., and Delarue, F. (2014) · 2014
Cited alongside, same era.
Mean Field Type Control with Congestion (II): An augmented Lagrangian method
Achdou, Y. and Laurière, M. (2016) · 2016
Cited alongside, same era.
Increasing the action gap: New operators for reinforcement learning
Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., and Munos, R. (2016) · 2016
Cited alongside, same era.
Mean-field-game model for botnet defense in cyber-security
Kolokoltsov, V. N. and Bensoussan, A. (2016) · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016) · 2016
Cited alongside, same era.
Two numerical approaches to stationary mean-field games
Almulla, N., Ferreira, R., and Gomes, D. (2017) · 2017
Cited alongside, same era.
On the properties of the softmax function with application in game theory and reinforcement learning
Original
Gao, B. and Pavel, L. (2017) · 2017
Cited alongside, same era.
Random measures, theory and applications
Kallenberg, O. (2017) · 2017
Cited alongside, same era.
Probabilistic theory of mean field games with applications. I
Carmona, R. and Delarue, F. (2018a)
Cited in the paper.

Then

The master equation and the convergence problem in mean field games
Cardaliaguet, P., Delarue, F., Lasry, J.-M., and Lions, P.-L. (2019) · 2019
Closest in time.
Actor-critic provably finds nash equilibria of linear-quadratic mean-field games
Fu, Z., Yang, Z., Chen, Y., and Wang, Z. (2019) · 2019
Closest in time.
Learning mean-field games
Guo, X., Hu, A., Xu, R., and Zhang, J. (2019) · 2019
Closest in time.
Reinforcement learning in stationary mean-field games
Subramanian, J. and Mahajan, A. (2019) · 2019
Closest in time.
Deep learning and stochastic mean-field control for a neural network model
Agram, N., Bakdi, A., and Oksendal, B. (2020) · 2020
Closest in time.
Policy optimization for linear-quadratic zero-sum mean-field type games
Carmona, R., Hamidouche, K., Laurière, M., and Tan, Z. (2020) · 2020
Closest in time.
On the convergence of model free learning in mean field games
Elie, R., Perolat, J., Laurière, M., Geist, M., and Pietquin, O. (2020) · 2020
Closest in time.
Deep learning methods for mean field control problems with delay
Fouque, J.-P. and Zhang, Z. (2020) · 2020
Closest in time.
Fictitious play for mean field games: Continuous time analysis and applications
Perrin, S., Pérolat, J., Laurière, M., Geist, M., Elie, R., and Pietquin, O. (2020) · 2020
Closest in time.
A machine learning framework for solving high-dimensional mean field game and mean field control problems
Ruthotto, L., Osher, S. J., Li, W., Nurbekyan, L., and Fung, S. W. (2020) · 2020
Closest in time.
Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games i: The ergodic case
Carmona, R. and Laurière, M. (2021) · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…