Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control

Understand

Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration in Reinforcement Learning (RL).

• However, state-of-the-art methods for continuous actions still suffer from high sample complexity requirements.
• Indeed, they either completely lack strategies for propagating the epistemic uncertainty throughout the updates, or they mix it with aleatoric uncertainty while learning the full return distribution (e.g., distributional RL).
• In this paper, we propose Wasserstein Actor-Critic (WAC), an actor-critic architecture inspired by the recent Wasserstein Q-Learning (WQL) \citep{wql}, that employs approximate Q-posteriors to represent the epistemic uncertainty and Wasserstein barycenters for uncertainty propagation across the state-action space.