A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces

Understand

We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space.

• The flow is a continuous-time analogue of a policy mirror descent method.
• We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy.
• Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation.