EXPO: Stable Reinforcement Learning with Expressive Policies

Understand

We study the problem of training and fine-tuning expressive policies with online reinforcement learning (RL) given an offline dataset.

• Training expressive policy classes with online RL present a unique challenge of stable value maximization.
• Unlike simpler Gaussian policies commonly used in online RL, expressive policies like diffusion and flow-matching policies are parameterized by a long denoising chain, which hinders stable gradient propagation from actions to policy parameters when optimizing against some value function.
• Our key insight is that we can address stable value maximization by avoiding direct optimization over value with the expressive policy and instead construct an on-the-fly RL policy to maximize Q-value.