Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost

Understand

In this paper, we consider model-free federated reinforcement learning for tabular episodic Markov decision processes.

• Under the coordination of a central server, multiple agents collaboratively explore the environment and learn an optimal policy without sharing their raw data.
• Despite recent advances in federated Q-learning algorithms achieving near-linear regret speedup with low communication cost, existing algorithms only attain suboptimal regrets compared to the information bound.
• We propose a novel model-free federated Q-learning algorithm, termed FedQ-Advantage.