Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

Two notions of correctness and their relation to testing

Timothy A. Budd and Dana Angluin. 1982 · 1982

Earlier work this paper cites.

Mutation Testing for the New Century

W Eric Wong, J R Horgan, Saul London, and Hira Agrawal. 2000 · 2000

Earlier work this paper cites.

Data mining: practical machine learning tools and techniques with Java implementations

Ian H Witten and Eibe Frank. 2002 · 2002

Earlier work this paper cites.

An analysis and survey of the development of mutation testing

Yue Jia and Mark Harman. 2010 · 2010

Earlier work this paper cites.

Is Operator-Based Mutant Selection Superior to Random Mutant Selection?. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 (Cape Town, South Africa) (ICSE ’10) . Association for Computing Machinery, New York, NY, USA, 435–444

Lu Zhang, Shan-Shan Hou, Jun-Jue Hu, Tao Xie, and Hong Mei. 2010 · 2010

Earlier work this paper cites.

Predicting folding free energy changes upon single point mutations

Zhe Zhang, Lin Wang, Yang Gao, Jie Zhang, Maxim Zhenirovskyy, and Emil Alexov. 2012 · 2012

Earlier work this paper cites.

Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation

Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. 2014 · 2013

Earlier work this paper cites.

Designing Deletion Mutation Operators. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation . 11–20

Marcio Eduardo Delamaro, Jeff Offutt, and Paul Ammann. 2014 · 2014

Earlier work this paper cites.

Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering . 654–665

René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014 · 2014

Earlier work this paper cites.

How hard does mutation analysis have to be, anyway?. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 216–227

Rahul Gopinath, Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. 2015 · 2015

Earlier work this paper cites.

Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , Vol. 1. IEEE, 936–946

Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015 · 2015

Earlier work this paper cites.

Mutation testing techniques: A comparative study. In 2016 international conference on engineering & MIS (ICEMIS) . IEEE, 1–9

Soukaina Hamimoune and Bouchaib Falah. 2016 · 2016

Earlier work this paper cites.

A Systematic Review of Mutation Testing Tools: A Survey

Pedro Delgado-Pérez, Inmaculada Medina-Bulo, and Antonio García-Domínguez. 2017 · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Earlier work this paper cites.

State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice . 163–171

Goran Petrović and Marko Ivanković. 2018 · 2018

Cited alongside, same era.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Cited alongside, same era.

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . 3539–3549

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018 · 2018

Cited alongside, same era.

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 2019

Cited alongside, same era.

Mutation Testing Advances: An Analysis and Survey

The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference . 10–19

James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022 · 2022

Later among the works it cites.

Can OpenAI’s codex fix bugs? an evaluation on QuixBugs. In Proceedings of the Third International Workshop on Automated Program Repair . 69–75

Julian Aron Prenner, Hlib Babii, and Romain Robbes. 2022 · 2022

Later among the works it cites.

Benchmarking Language Models for Code Syntax Understanding

Original

Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, and Dawn Song. 2022 · 2022

Later among the works it cites.

Probing pretrained models of source code

Original

Sergey Troshin and Nadezhda Chirkova. 2022 · 2022

Later among the works it cites.