Comparing Humans and Models on a Similar Scale: Towards Cognitive Gender Bias Evaluation in Coreference Resolution

Understand

Spurious correlations were found to be an important factor explaining model performance in various NLP tasks (e.g., gender or racial artifacts), often considered to be ''shortcuts'' to the actual task.

• However, humans tend to similarly make quick (and sometimes wrong) predictions based on societal and cognitive presuppositions.
• In this work we address the question: can we quantify the extent to which model biases reflect human behaviour? Answering this question will help shed light on model performance and provide meaningful comparisons against humans.
• We approach this question through the lens of the dual-process theory for human decision-making.