MiniLLM: On-Policy Distillation of Large Language Models

Understand

Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs).

• However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT.
• How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs.
• In this work, we propose a KD approach that distills LLMs into smaller language models.