CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

Understand

This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO).

• GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question.
• Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training -- their contribution depends on their relative advantage.
• To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates.