OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Understand

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a.

• instruction-tuning, improves their zero and few-shot generalization to unseen tasks.
• However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process.
• These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves.