Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning

Understand

Recent advances in reinforcement learning with verifiable rewards (RLVR) show that large language models enhance their reasoning abilities when trained with verifiable signals.

• However, due to reward sparsity, effectiveness depends heavily on selecting samples of appropriate difficulty.
• In this work, we present a formal analysis of online difficulty-aware filtering and establish its theoretical foundations.
• We show that expected policy improvement is lower-bounded by the variance of task-level success probabilities, implying that selecting tasks of intermediate difficulty maximizes learning efficiency.