$\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

Understand

Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input.

• Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $\pi_0$, $\pi_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching.
• We address this challenge with $\pi_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation.
• (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration.