Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

Understand

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL).

• At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy.
• We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets.
• To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement.