Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language

Understand

This study introduces a prescriptive annotation benchmark grounded in humanities research to ensure consistent, unbiased labeling of offensive language, particularly for casual and non-mainstream language uses.

• We contribute two newly annotated datasets that achieve higher inter-annotator agreement between human and language model (LLM) annotations compared to original datasets based on descriptive instructions.
• Our experiments show that LLMs can serve as effective alternatives when professional annotators are unavailable.
• Moreover, smaller models fine-tuned on multi-source LLM-annotated data outperform models trained on larger, single-source human-annotated datasets.