Measuring Normative and Descriptive Biases in Language Models Using Census Data

Understand

We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models.

• Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality.
• In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions.
• To this end, we use official demographic information about gender--occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States.