Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation

Understand

Large language models (LLMs) are increasingly trained with reinforcement learning from verifiable rewards (RLVR), yet real-world deployment demands models that can self-improve without labels or external judges.

• Existing self-improvement approaches primarily rely on self-confirmation signals (e.g., confidence, entropy, or consistency) to generate rewards.
• This reliance drives models toward over-confident, majority-favored solutions, causing an entropy collapse that degrades pass@n and reasoning complexity.
• To address this, we propose EVOL-RL, a label-free framework that mirrors the evolutionary principle of balancing selection with variation.