2024

Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM

Ryan, Gabriel, Jain, Siddhartha, Shang, Mingyue et al.

Understand

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage.

Recent works using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance.
As a result LLM-generated testsuites still suffer from low coverage.
In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation.

Built on

Symbolic execution and program testing
James C King. 1976 · 1976
Earlier work this paper cites.
A Complexity Measure
Thomas J. McCabe. 1976 · 1976
Earlier work this paper cites.
Korat: Automated testing based on Java predicates
Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002 · 2002
Earlier work this paper cites.
The economic impacts of inadequate infrastructure for software testing
Strategic Planning. 2002 · 2002
Earlier work this paper cites.
DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation . 213–223
Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005 · 2005
Earlier work this paper cites.
CUTE: A concolic unit testing engine for C
Koushik Sen, Darko Marinov, and Gul Agha. 2005 · 2005
Earlier work this paper cites.
Compositional dynamic test generation. In Proceedings of the 34th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages . 47–54
Patrice Godefroid. 2007 · 2007
Earlier work this paper cites.
Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion . 815–816
Carlos Pacheco and Michael D Ernst. 2007 · 2007
Earlier work this paper cites.
Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07) . IEEE, 75–84
Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007 · 2007
Earlier work this paper cites.
Pex–white box test generation for. net. In International conference on tests and proofs . Springer, 134–153
Nikolai Tillmann and Jonathan De Halleux. 2008 · 2008
Earlier work this paper cites.
Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering . 416–419
Gordon Fraser and Andrea Arcuri. 2011 · 2011
Earlier work this paper cites.
Regression testing minimization, selection and prioritization: a survey
Shin Yoo and Mark Harman. 2012 · 2012
Earlier work this paper cites.
Evosuite: On the challenges of test case generation in the real world. In 2013 IEEE sixth international conference on software testing, verification and validation . IEEE, 362–369
Gordon Fraser and Andrea Arcuri. 2013 · 2013
Earlier work this paper cites.
A survey on unit testing practices and problems. In 2014 IEEE 25th International Symposium on Software Reliability Engineering . IEEE, 201–211
Ermira Daka and Gordon Fraser. 2014 · 2014
Earlier work this paper cites.

Similar

A large-scale evaluation of automated unit test generation using evosuite
Gordon Fraser and Andrea Arcuri. 2014 · 2014
Cited alongside, same era.
Suggesting accurate method and class names. In Proceedings of the 2015 10th joint meeting on foundations of software engineering . 38–49
Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015 · 2015
Cited alongside, same era.
1600 faults in 100 projects: automatically finding faults while achieving high coverage with evosuite
Gordon Fraser and Andrea Arcuri. 2015 · 2015
Cited alongside, same era.
A survey of symbolic execution techniques
Roberto Baldoni, Emilio Coppa, Daniele Cono D’elia, Camil Demetrescu, and Irene Finocchi. 2018 · 2018
Cited alongside, same era.
A Transformer-based Approach for Source Code Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 4998–5007
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020 · 2020
Cited alongside, same era.
Automated unit test generation for python. In Search-Based Software Engineering: 12th International Symposium, SSBSE 2020, Bari, Italy, October 7–8, 2020, Proceedings 12 . Springer, 9–24
Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2020 · 2020
Cited alongside, same era.
Unit test case generation with transformers and focal context
Original
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020 · 2020
Cited alongside, same era.
BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020) . Association for Computing Machinery, New York, NY, USA, 1556–1560
Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. 2020 · 2020
Cited alongside, same era.
Program synthesis with large language models
Original
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al · 2021
Cited alongside, same era.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021
Cited alongside, same era.
Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code
Original
Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel. 2022 · 2022
Cited alongside, same era.
Toga: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering . 2130–2141
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022 · 2022
Cited alongside, same era.

Then

Emergent abilities of large language models
Original
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al · 2022
Later among the works it cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Later among the works it cites.
A3Test: Assertion-Augmented Automated Test Case Generation
Original
Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2023 · 2023
Later among the works it cites.
Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing
Original
Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Desmarais. 2023 · 2023
Later among the works it cites.
Automated Test Case Generation Using Code Models and Domain Adaptation
Original
Sepehr Hashtroudi, Jiho Shin, Hadi Hemmati, and Song Wang. 2023 · 2023
Later among the works it cites.
CODAMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In International conference on software engineering (ICSE)
Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023 · 2023
Later among the works it cites.
Finding Failure-Inducing Test Cases with ChatGPT
Original
Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, and Shing-Chi Cheung. 2023 · 2023
Later among the works it cites.
Codegen2: Lessons for training llms on programming and natural languages
Original
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023 · 2023
Later among the works it cites.
GPT-4 Technical Report
Original
OpenAI. 2023 · 2023
Later among the works it cites.
Adaptive test generation using a large language model
Original
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023a · 2023
Later among the works it cites.
An empirical evaluation of using large language models for automated unit test generation
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023b · 2023
Later among the works it cites.
ChatUniTest: a ChatGPT-based automated unit test generation tool
Original
Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023 · 2023
Later among the works it cites.
Tree of thoughts: Deliberate problem solving with large language models
Original
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023 · 2023
Later among the works it cites.
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
Original
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023 · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…