Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency

Understand

With the availability of various instruction datasets, a pivotal challenge is how to effectively select and integrate these instructions to fine-tune large language models (LLMs).

• Previous research mainly focuses on selecting individual high-quality instructions.
• However, these works overlooked the joint interactions and dependencies between different categories of instructions, leading to suboptimal selection strategies.
• Moreover, the nature of these interaction patterns remains largely unexplored, let alone optimize the instruction set with regard to them.