BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models

Understand

Pretrained Language Models (PLMs) harbor inherent social biases that can result in harmful real-world implications.

• Such social biases are measured through the probability values that PLMs output for different social groups and attributes appearing in a set of test sentences.
• However, bias testing is currently cumbersome since the test sentences are generated either from a limited set of manual templates or need expensive crowd-sourcing.
• We instead propose using ChatGPT for the controllable generation of test sentences, given any arbitrary user-specified combination of social groups and attributes appearing in the test sentences.