COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models

Understand

Large Language Models (LLMs) often inherit biases from the web data they are trained on, which contains stereotypes and prejudices.

• Current methods for evaluating and mitigating these biases rely on bias-benchmark datasets.
• These benchmarks measure bias by observing an LLM's behavior on biased statements.
• However, these statements lack contextual considerations of the situations they try to present.