Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing

Understand

Bias auditing of language models (LMs) has received considerable attention as LMs are becoming widespread.

• As such, several benchmarks for bias auditing have been proposed.
• At the same time, the rapid evolution of LMs can make these benchmarks irrelevant in no time.
• Bias auditing is further complicated by LM brittleness: when a presumably biased outcome is observed, is it due to model bias or model brittleness? We propose enlisting the models themselves to help construct bias auditing datasets that remain challenging, and introduce bias measures that distinguish between different types of model errors.