Datasets for Fairness in Language Models: An In-Depth Survey

Understand

Despite the growing reliance on fairness benchmarks to evaluate language models, the datasets that underpin these benchmarks remain critically underexamined.

• This survey addresses that overlooked foundation by offering a comprehensive analysis of the most widely used fairness datasets in language model research.
• To ground this analysis, we characterize each dataset across key dimensions, including provenance, demographic scope, annotation design, and intended use, revealing the assumptions and limitations baked into current evaluation practices.
• Building on this foundation, we propose a unified evaluation framework that surfaces consistent patterns of demographic disparities across benchmarks and scoring metrics.