MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

Understand

Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes.

• While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages.
• There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user.
• Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy.