LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

Understand

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities.

• However, these models are resource-intensive.
• To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones.
• To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions.