2023

She Elicits Requirements and He Tests: Software Engineering Gender Bias in Large Language Models

Treude, Christoph, Hata, Hideaki

Understand

Implicit gender bias in software development is a well-documented issue, such as the association of technical roles with men.

To address this bias, it is important to understand it in more detail.
This study uses data mining techniques to investigate the extent to which 56 tasks related to software development, such as assigning GitHub issues and testing, are affected by implicit gender bias embedded in large language models.
We systematically translated each task from English into a genderless language and back, and investigated the pronouns associated with each task.

Built on

R. L. Glass, I. Vessey, and S. A. Conger, “Software tasks: Intellectual or clerical?” Information & Management , vol. 23, no. 4, pp. 183–191, 1992
1992
Earlier work this paper cites.
P. M. Fishman, Interaction: The work women do . Springer, 1997
1997
Earlier work this paper cites.
A. E. Milewski, “Global and task effects in information-seeking among software engineers,” Empirical Software Engineering , vol. 12, no. 3, pp. 311–326, 2007
2007
Earlier work this paper cites.
S. Brough, A. E. Bauer, K. Brooks, A. Hopkins, and S. Maddison, “Women in astronomy workshop report,” arXiv preprint arXiv:1106.6094 , 2011
Original
2011
Earlier work this paper cites.
B. Vasilescu, A. Capiluppi, and A. Serebrenik, “Gender, representation and online participation: A quantitative study of StackOverflow,” in Proceedings of the International Conference on Social Informatics , 2012, pp. 332–338
2012
Earlier work this paper cites.
V. Kuechler, C. Gilbertson, and C. Jensen, “Gender differences in early free and open source software joining process,” in Proceedings of Open Source Systems: Long-Term Sustainability , I. Hammouda, B. Lundell, T. Mikkonen, and W. Scacchi, Eds., 2012, pp. 78–93
2012
Earlier work this paper cites.
A. Murgia, P. Tourani, B. Adams, and M. Ortu, “Do developers feel emotions? An exploratory analysis of emotions in software artifacts,” in Proceedings of the 11th Working Conference on Mining Software Repositories , 2014, pp. 262–271
2014
Earlier work this paper cites.
B. Vasilescu, D. Posnett, B. Ray, M. G. van den Brand, A. Serebrenik, P. Devanbu, and V. Filkov, “Gender and tenure diversity in GitHub teams,” in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , 2015, pp. 3789–3798
2015
Earlier work this paper cites.
D. Ford and C. Parnin, “Exploring causes of frustration for software developers,” in Proceedings of the 8th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering . IEEE, 2015, pp. 115–116
2015
Earlier work this paper cites.
J. Terrell, A. Kofink, J. Middleton, C. Rainear, E. Murphy-Hill, C. Parnin, and J. Stallings, “Gender differences and bias in open source: Pull request acceptance of women versus men,” PeerJ Computer Science , vol. 3, p. e111, 2017
2017
Earlier work this paper cites.

Similar

A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science , vol. 356, no. 6334, pp. 183–186, 2017
2017
Cited alongside, same era.
S. A. Licorish and S. G. MacDonell, “Exploring software developers’ work practices: Task differences, participation, engagement, and speed of task resolution,” Information & Management , vol. 54, no. 3, pp. 364–382, 2017
2017
Cited alongside, same era.
D. Graziotin, F. Fagerholm, X. Wang, and P. Abrahamsson, “On the unhappiness of software developers,” in Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering , 2017, pp. 324–333
2017
Cited alongside, same era.
S. Prabhumoye, Y. Tsvetkov, R. Salakhutdinov, and A. W. Black, “Style transfer through back-translation,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018, pp. 866–876
2018
Cited alongside, same era.
Y. Wang and D. Redmiles, “Implicit gender biases in professional software development: An empirical study,” in Proceedings of the 41st ACM/IEEE International Conference on Software Engineering: Software Engineering in Society . IEEE, 2019, pp. 1–10
2019
Cited alongside, same era.
S. Bordia and S. Bowman, “Identifying and reducing gender bias in word-level language models,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop , 2019, pp. 7–15
2019
Cited alongside, same era.
N. Imtiaz, J. Middleton, J. Chakraborty, N. Robson, G. Bai, and E. Murphy-Hill, “Investigating the effects of gender bias on GitHub,” in Proceedings of the 41st International Conference on Software Engineering , ser. ICSE ’19. IEEE Press, 2019, p. 700–711. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00079
2019
Cited alongside, same era.
A. N. Meyer, E. T. Barr, C. Bird, and T. Zimmermann, “Today was a good day: The daily life of software developers,” IEEE Transactions on Software Engineering , vol. 47, no. 5, pp. 863–880, 2019
2019
Cited alongside, same era.
P.-S. Huang, H. Zhang, R. Jiang, R. Stanforth, J. Welbl, J. Rae, V. Maini, D. Yogatama, and P. Kohli, “Reducing sentiment bias in language models via counterfactual evaluation,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , 2020, pp. 65–83
2020
Cited alongside, same era.
K. Madampe, R. Hoda, and P. Singh, “Towards understanding emotional response to requirements changes in Agile teams,” in Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering: New Ideas and Emerging Results , 2020, pp. 37–40
2020
Cited alongside, same era.

Then

M. P. Robillard, “Turnover-induced knowledge loss in practice,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 1292–1302
2021
Later among the works it cites.
P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, “Towards understanding and mitigating social biases in language models,” in Proceedings of the International Conference on Machine Learning . PMLR, 2021, pp. 6565–6576
2021
Later among the works it cites.
R. Liu, C. Jia, J. Wei, G. Xu, L. Wang, and S. Vosoughi, “Mitigating political bias in language models through reinforced calibration,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 17, 2021, pp. 14 857–14 866
2021
Later among the works it cites.
N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” ACM Computing Surveys , vol. 54, no. 6, jul 2021. [Online]. Available: https://doi.org/10.1145/3457607
2021
Later among the works it cites.
R. Garcia, C.-J. Liao, A. Pearce, and C. Treude, “Gender influence on communication initiated within student teams,” in Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 , 2022, pp. 432–438
2022
Later among the works it cites.
Z. Masood, R. Hoda, K. Blincoe, and D. Damian, “Like, dislike, or just do it? how developers approach software development tasks,” Information and Software Technology , p. 106963, 2022
2022
Later among the works it cites.
G. A. A. Prana, D. Ford, A. Rastogi, D. Lo, R. Purandare, and N. Nagappan, “Including everyone, everywhere: Understanding opportunities and challenges of geographic gender-inclusion in OSS,” IEEE Transactions on Software Engineering , vol. 48, no. 9, pp. 3394–3409, 2022
2022
Later among the works it cites.
H. S. Padala, C. Mendez, F. Fronchetti, I. Steinmacher, Z. Steine-Hanson, C. Hilderbrand, A. Horvath, C. Hill, L. Simpson, M. Burnett, M. Gerosa, and A. Sarma, “How gender-biased tools shape newcomer experiences in OSS projects,” IEEE Transactions on Software Engineering , vol. 48, no. 1, pp. 241–259, 2022
2022
Later among the works it cites.
D. Rossi and S. Zacchiroli, “Worldwide gender differences in public code contributions: And how they have been affected by the COVID-19 pandemic,” in Proceedings of the 44th ACM/IEEE International Conference on Software Engineering: Software Engineering in Society , 2022, p. 172–183
2022
Later among the works it cites.
E. A. Renström, A. Lindqvist, G. Akbas, L. Hekanaho, and M. G. Sendén, “Are gender-neutral pronouns really neutral? Testing a male bias in the grammatical genderless languages Turkish and Finnish,” Journal of Language and Social Psychology , p. 0261927X221146229, 2022
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…