Distilling LLM Agent into Small Models with Retrieval and Code Tools

Understand

Large language models (LLMs) excel at complex reasoning tasks but remain computationally expensive, limiting their practical deployment.

• To address this, recent works have focused on distilling reasoning capabilities into smaller language models (sLMs) using chain-of-thought (CoT) traces from teacher LLMs.
• However, this approach struggles in scenarios requiring rare factual knowledge or precise computation, where sLMs often hallucinate due to limited capability.
• In this work, we propose Agent Distillation, a framework for transferring not only reasoning capability but full task-solving behavior from LLM-based agents into sLMs with retrieval and code tools.