Fetching the paper…

Reading the bibliography…

2019

Limitations of Pinned AUC for Measuring Unintended Bias

Borkan, Daniel, Dixon, Lucas, Li, John et al.

Original
PDF
Post on X

Understand

This report examines the Pinned AUC metric introduced and highlights some of its limitations.

Pinned AUC provides a threshold-agnostic measure of unintended bias in a classification model, inspired by the ROC-AUC metric.
However, as we highlight in this report, there are ways that the metric can obscure different kinds of unintended biases when the underlying class distributions on which bias is being measured are not carefully controlled.

Limitations of Pinned AUC for Measuring Unintended Bias · Around

Built on

Perspective api, 2017
Jigsaw · 2017
Earlier work this paper cites.
Measuring and mitigating unintended bias in text classification
L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman · 2018
Earlier work this paper cites.

Similar

Unintended bias and names of frequently targeted groups, 2018
Lucy Vasserman, John Li, CJ Adams, Lucas Dixon · 2018
Cited alongside, same era.
Areas beneath the relative operating characteristics (roc) and relative operating levels (rol) curves: Statistical significance and interpretation
Original
S. J. Mason and N. E. Graham
Cited in the paper.

Then

Nuanced metrics for measuring unintended bias with real data for text classification
D. Borkan, L. Dixon, J. Sorensen, N. Thain, and L. Vasserman · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…