How to Measure Gender Bias in Machine Translation: Optimal Translators, Multiple Reference Points

Understand

In this paper, as a case study, we present a systematic study of gender bias in machine translation with Google Translate.

• We translated sentences containing names of occupations from Hungarian, a language with gender-neutral pronouns, into English.
• Our aim was to present a fair measure for bias by comparing the translations to an optimal non-biased translator.
• When assessing bias, we used the following reference points: (1) the distribution of men and women among occupations in both the source and the target language countries, as well as (2) the results of a Hungarian survey that examined if certain jobs are generally perceived as feminine or masculine.