AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

Understand

Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks.

• Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems.
• Most existing large-scale RL systems for LLMs are synchronous, alternating generation and training in a batch setting where rollouts in each training batch are generated by the same model.
• This approach stabilizes RL training but suffers from severe system-level inefficiency: generation must wait until the longest output in the batch is completed before model updates, resulting in GPU underutilization.