Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

Understand

Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs).

• However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures.
• We show that plateaued RL models can successfully refine failed solutions when given natural language critiques.
• Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization.