Fetching the paper…
Reading the bibliography…
While knowledge distillation has become a mature field for compressing large language models (LLMs) into smaller ones by aligning their outputs or internal representations, the distillation of LLM-based agents, which involve planning, memory, and tool use, remains relatively underexplored.
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf · 2019
Earlier work this paper cites.
Patient knowledge distillation for bert model compression
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu · 2019
Earlier work this paper cites.
Tinybert: Distilling bert for natural language understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu · 2019
Earlier work this paper cites.
Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou · 2020
Earlier work this paper cites.
Mobilebert: a compact task-agnostic bert for resource-limited devices
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou · 2020
Earlier work this paper cites.
Pathvqa: 30000+ questions for medical visual question answering
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie · 2020
Earlier work this paper cites.
Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu · 2021
Earlier work this paper cites.
Large language models are reasoning teachers
Namgyu Ho, Laura Schmid, and Se-Young Yun · 2022
Earlier work this paper cites.
Explanations from large language models make small reasoners better
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al · 2022
Earlier work this paper cites.
Disco: Distilling counterfactuals with large language models
Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson · 2022
Earlier work this paper cites.
Learning by distilling context
Charlie Snell, Dan Klein, and Ruiqi Zhong · 2022
Earlier work this paper cites.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown · 2022
Earlier work this paper cites.
Gkd: A general knowledge distillation framework for large-scale pre-trained language model
Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang, Hongyin Tang, Keqing He, Jiahao Liu, Jingang Wang, Shu Zhao, et al · 2023
Earlier work this paper cites.
Distilling reasoning capabilities into smaller language models
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan · 2023
Earlier work this paper cites.
Symbolic chain-of-thought distillation: Small models can also" think" step-by-step
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi · 2023
Earlier work this paper cites.
Alpaca: A strong, replicable instruction-following model
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto · 2023
Earlier work this paper cites.
Orca: Progressive learning from complex explanation traces of gpt-4
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah · 2023
Earlier work this paper cites.
Scott: Self-consistent chain-of-thought distillation
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren · 2023
Cited alongside, same era.
Specializing smaller language models towards multi-step reasoning
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot · 2023
Cited alongside, same era.
Autoagents: A framework for automatic agent generation
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi · 2023
Cited alongside, same era.
Agentcoder: Multi-agent-based code generation with iterative testing and optimisation
Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui · 2023
Cited alongside, same era.
Keypoint-based progressive chain-of-thought distillation for llms
Structured agent distillation for large language model
Jun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Tianqi Li, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Pu Zhao, et al · 2025
Closest in time.
Distilling llm agent into small models with retrieval and code tools
Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, and Sung Ju Hwang · 2025
Closest in time.
Introducing the model context protocol
Anthropic · 2025
Closest in time.
Model context protocol (mcp): Landscape, security threats, and future research directions
Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang · 2025
Closest in time.
Mcip: Protecting mcp safety via model contextual integrity protocol
Huihao Jing, Haoran Li, Wenbin Hu, Qi Hu, Heli Xu, Tianshu Chu, Peizhao Hu, and Yangqiu Song · 2025
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, and Guoren Wang · 2024
Cited alongside, same era.
Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal · 2024
Cited alongside, same era.
Sub-goal distillation: A method to improve small language agents
Maryam Hashemzadeh, Elias Stengel-Eskin, Sarath Chandar, and Marc-Alexandre Cote · 2024
Cited alongside, same era.
Learning to maximize mutual information for chain-of-thought distillation
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding · 2024
Cited alongside, same era.
Efficient llm context distillation
Rajesh Upadhayayaya, Zachary Smith, Chritopher Kottmyer, and Manish Raj Osti · 2024
Cited alongside, same era.
In-context learning distillation for efficient few-shot fine-tuning
Yifei Duan, Liu Li, Zirui Zhai, and Jinxia Yao · 2024
Cited alongside, same era.
Knowledge distillation from language-oriented to emergent communication for multi-agent remote control
Yongjun Kim, Sejin Seo, Jihong Park, Mehdi Bennis, Seong-Lyun Kim, and Junil Choi · 2024
Cited alongside, same era.
Embodied cot distillation from llm to off-the-shelf agents
Wonje Choi, Woo Kyung Kim, Minjong Yoo, and Honguk Woo · 2024
Cited alongside, same era.
Closest in time.
Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, et al · 2025
Closest in time.
Supercorrect: Advancing small llm reasoning with thought template distillation and self-correction
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan · 2025
Closest in time.
Xiao Chen, Sihang Zhou, Ke Liang, Xiaoyu Sun, and Xinwang Liu · 2025
Closest in time.
Reasonflux: Hierarchical llm reasoning via scaling thought templates
Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang · 2025
Closest in time.
Unicott: A unified framework for structural chain-of-thought distillation
Xianwei Zhuang, Zhihong Zhu, Zhichang Wang, Xuxin Cheng, and Yuexian Zou · 2025
Closest in time.
Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, et al · 2025
Closest in time.
Chemagent: Self-updating library in large language models improves chemical reasoning
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, et al · 2025
Closest in time.
On path to multimodal historical reasoning: Histbench and histagent
Jiahao Qiu, Fulian Xiao, Yimin Wang, Yuchen Mao, Yijia Chen, Xinzhe Juan, Siran Wang, Xuan Qi, Tongcheng Zhang, Zixin Yao, et al · 2025
Closest in time.
Emoagent: Assessing and safeguarding human-ai interaction for mental health safety
Jiahao Qiu, Yinghui He, Xinzhe Juan, Yimin Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, and Mengdi Wang · 2025
Closest in time.
‘smolagents‘: a smol library to build great agentic systems
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki · 2025
Closest in time.
Math twenty four (24 -
Nathan Lile · 2025
Closest in time.
Octotools: An agentic framework with extensible tools for complex reasoning
Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou · 2025
Closest in time.