Datasets for Large Language Models: A Comprehensive Survey

Understand

This paper embarks on an exploration into the Large Language Model (LLM) datasets, which play a crucial role in the remarkable advancements of LLMs.

• The datasets serve as the foundational infrastructure analogous to a root system that sustains and nurtures the development of LLMs.
• Consequently, examination of these datasets emerges as a critical topic in research.
• In order to address the current lack of a comprehensive overview and thorough analysis of LLM datasets, and to gain insights into their current status and future trends, this survey consolidates and categorizes the fundamental aspects of LLM datasets from five perspectives: (1) Pre-training Corpora; (2) Instruction Fine-tuning Datasets; (3) Preference Datasets; (4) Evaluation Datasets; (5) Traditional Natural Language Processing (NLP) Datasets.