Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

Understand

Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases.

• Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures.
• However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators.
• To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples.