UnQovering Stereotyping Biases via Underspecified Questions

Understand

While language embeddings have been shown to have stereotyping biases, how these biases affect downstream question answering (QA) models remains unexplored.

• We present UNQOVER, a general framework to probe and quantify biases through underspecified questions.
• We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors: positional dependence and question independence.
• We design a formalism that isolates the aforementioned errors.