MoDS: Model-oriented Data Selection for Instruction Tuning

Understand

Instruction tuning has become the de facto method to equip large language models (LLMs) with the ability of following user instructions.

• Usually, hundreds of thousands or millions of instruction-following pairs are employed to fine-tune the foundation LLMs.
• Recently, some studies show that a small number of high-quality instruction data is enough.
• However, how to select appropriate instruction data for a given LLM is still an open problem.