FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks

Understand

Large language models (LLMs) are increasingly used in high-stakes decisions such as hiring and college admissions, making their social bias a critical concern.

• While LLMs are trained to refuse explicitly biased requests, bias can be leaked implicitly during LLM planning and reasoning process.
• As code becomes the primary medium for LLM internal logic-writing, we introduce FairCoder, a benchmark that frames decision-making as coding tasks to systematically probe LLM bias across employment, education, and healthcare domains, covering multiple fairness definitions.
• Considering that existing metrics may fail when LLMs frequently refuse the request, we propose FairScore, a metric that jointly captures refusal behavior and group-level outcome diversity.