RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

Understand

Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities.

• To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through sophisticated logical structures without modifying LLMs' parameters.
• However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability.
• To improve this, we propose RL-of-Thoughts (RLoT), where we train a lightweight navigator model with reinforcement learning (RL) to adaptively enhance LLM reasoning at inference time.