A Statistical Analysis of Polyak-Ruppert Averaged Q-learning

Understand

We study Q-learning with Polyak-Ruppert averaging in a discounted Markov decision process in synchronous and tabular settings.

• Under a Lipschitz condition, we establish a functional central limit theorem for the averaged iteration $\bar{\boldsymbol{Q}}_T$ and show that its standardized partial-sum process converges weakly to a rescaled Brownian motion.
• The functional central limit theorem implies a fully online inference method for reinforcement learning.
• Furthermore, we show that $\bar{\boldsymbol{Q}}_T$ is the regular asymptotically linear (RAL) estimator for the optimal Q-value function $\boldsymbol{Q}^*$ that has the most efficient influence function.