Beyond Imitation: Reinforcement Learning for Active Latent Planning

Understand

Aiming at efficient and dense chain-of-thought (CoT) reasoning, latent reasoning methods fine-tune Large Language Models (LLMs) to substitute discrete language tokens with continuous latent tokens.

• These methods consume fewer tokens compared to the conventional language CoT reasoning and have the potential to plan in a dense latent space.
• However, current latent tokens are generally supervised based on imitating language labels.
• Considering that there can be multiple equivalent but diverse CoT labels for a question, passively imitating an arbitrary one may lead to inferior latent token representations and latent reasoning policies, undermining the potential planning ability and resulting in clear gaps between training and testing.