Flow-Based Policy for Online Reinforcement Learning

Understand

We present \textbf{FlowRL}, a novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein-2-regularized optimization.

• We argue that in addition to training signals, enhancing the expressiveness of the policy class is crucial for the performance gains in RL.
• Flow-based generative models offer such potential, excelling at capturing complex, multimodal action distributions.
• However, their direct application in online RL is challenging due to a fundamental objective mismatch: standard flow training optimizes for static data imitation, while RL requires value-based policy optimization through a dynamic buffer, leading to difficult optimization landscapes.