An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation

B. P. Miller, L. Fredriksen, and B. So, “An empirical study of the reliability of UNIX utilities,” Commun. ACM , vol. 33, no. 12, pp. 32–44, 1990. [Online]. Available: https://doi.org/10.1145/96267.96279

1990

Earlier work this paper cites.

N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.” Psychological bulletin , vol. 114, no. 3, p. 494, 1993

1993

Earlier work this paper cites.

G. Myers, “A fast bit-vector algorithm for approximate string matching based on dynamic programming,” J. ACM , vol. 46, no. 3, p. 395–415, may 1999. [Online]. Available: https://doi.org/10.1145/316542.316550

1999

Earlier work this paper cites.

K. Beck, Extreme Programming Explained: Embrace Change . Addison-Wesley, 2000

2000

Earlier work this paper cites.

S. Schleimer, D. S. Wilkerson, and A. Aiken, “Winnowing: Local algorithms for document fingerprinting,” in Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data , ser. SIGMOD ’03. New York, NY, USA: Association for Computing Machinery, 2003, p. 76–85. [Online]. Available: https://doi.org/10.1145/872757.872770

2003

Earlier work this paper cites.

C. Csallner and Y. Smaragdakis, “JCrasher: an automatic robustness tester for Java,” Software Practice and Experience , vol. 34, no. 11, pp. 1025–1050, 2004

2004

Earlier work this paper cites.

P. Godefroid, N. Klarlund, and K. Sen, “DART: directed automated random testing,” in Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, Chicago, IL, USA, June 12-15, 2005 , V. Sarkar and M. W. Hall, Eds. ACM, 2005, pp. 213–223. [Online]. Available: https://doi.org/10.1145/1065010.1065036

2005

Earlier work this paper cites.

K. Sen, D. Marinov, and G. Agha, “CUTE: a concolic unit testing engine for C,” in Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon, Portugal, September 5-9, 2005 , M. Wermelinger and H. C. Gall, Eds. ACM, 2005, pp. 263–272. [Online]. Available: https://doi.org/10.1145/1081706.1081750

2005

Earlier work this paper cites.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020. [Online]. Available: https://arxiv.org/abs/2005.14165

Original

2005

Earlier work this paper cites.

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler, “EXE: automatically generating inputs of death,” in Proceedings of the 13th ACM Conference on Computer and Communications Security, CCS 2006, Alexandria, VA, USA, October 30 - November 3, 2006 , A. Juels, R. N. Wright, and S. D. C. di Vimercati, Eds. ACM, 2006, pp. 322–335. [Online]. Available: https://doi.org/10.1145/1180405.1180445

2006

Earlier work this paper cites.

C. Pacheco and M. D. Ernst, “Randoop: Feedback-directed random testing for java,” in Companion to the 22Nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion , ser. OOPSLA ’07. New York, NY, USA: ACM, 2007, pp. 815–816. [Online]. Available: http://doi.acm.org/10.1145/1297846.1297902

2007

Earlier work this paper cites.

C. Pacheco, S. K. Lahiri, and T. Ball, “Finding errors in .net with feedback-directed random testing,” in Proceedings of the 2008 International Symposium on Software Testing and Analysis , ser. ISSTA ’08. New York, NY, USA: ACM, 2008, pp. 87–96. [Online]. Available: http://doi.acm.org/10.1145/1390630.1390643

2008

Earlier work this paper cites.

P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCamant, and D. Song, “A symbolic execution framework for JavaScript,” in 31st IEEE Symposium on Security and Privacy, S&P 2010, 16-19 May 2010, Berleley/Oakland, California, USA . IEEE Computer Society, 2010, pp. 513–528. [Online]. Available: https://doi.org/10.1109/SP.2010.38

2010

Earlier work this paper cites.

G. Fraser and A. Arcuri, “Evolutionary generation of whole test suites,” in Proceedings of the 11th International Conference on Quality Software, QSIC 2011, Madrid, Spain, July 13-14, 2011 , M. Núñez, R. M. Hierons, and M. G. Merayo, Eds. IEEE Computer Society, 2011, pp. 31–40. [Online]. Available: https://doi.org/10.1109/QSIC.2011.19

2011

Earlier work this paper cites.

——, “EvoSuite: automatic test suite generation for object-oriented software,” in SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011 , T. Gyimóthy and A. Zeller, Eds. ACM, 2011, pp. 416–419. [Online]. Available: https://doi.org/10.1145/2025113.2025179

2011

Earlier work this paper cites.

D. Schuler and A. Zeller, “Assessing oracle quality with checked coverage,” in Fourth IEEE International Conference on Software Testing, Verification and Validation , 2011, pp. 90–99

2011

Earlier work this paper cites.

P. McMinn, “Search-based software testing: Past, present and future,” in Fourth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Berlin, Germany, 21-25 March, 2011, Workshop Proceedings . IEEE Computer Society, 2011, pp. 153–163. [Online]. Available: https://doi.org/10.1109/ICSTW.2011.100

2011

Earlier work this paper cites.

S. Artzi, J. Dolby, S. H. Jensen, A. Møller, and F. Tip, “A framework for automated testing of JavaScript web applications,” in Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu , HI, USA, May 21-28, 2011 , R. N. Taylor, H. C. Gall, and N. Medvidovic, Eds. ACM, 2011, pp. 571–580. [Online]. Available: https://doi.org/10.1145/1985793.1985871

2011

Earlier work this paper cites.

A. Marchetto and P. Tonella, “Using search-based algorithms for Ajax event sequence generation during testing,” Empir. Softw. Eng. , vol. 16, no. 1, pp. 103–140, 2011. [Online]. Available: https://doi.org/10.1007/s10664-010-9149-1

2011

Earlier work this paper cites.

S. Yoo and M. Harman, “Regression testing minimization, selection and prioritization: a survey,” Software testing, verification and reliability , vol. 22, no. 2, pp. 67–120, 2012

2012

Earlier work this paper cites.

S. Mirshokraie and A. Mesbah, “JSART: JavaScript assertion-based regression testing,” in Web Engineering - 12th International Conference, ICWE 2012, Berlin, Germany, July 23-27, 2012. Proceedings , ser. Lecture Notes in Computer Science, M. Brambilla, T. Tokuda, and R. Tolksdorf, Eds., vol. 7387. Springer, 2012, pp. 238–252. [Online]. Available: https://doi.org/10.1007/978-3-642-31753-8_18

2012

Earlier work this paper cites.

S. Mirshokraie, A. Mesbah, and K. Pattabiraman, “PYTHIA: generating test cases with oracles for JavaScript applications,” in 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11-15, 2013 , E. Denney, T. Bultan, and A. Zeller, Eds. IEEE, 2013, pp. 610–615. [Online]. Available: https://doi.org/10.1109/ASE.2013.6693121

2013

Earlier work this paper cites.

E. Daka and G. Fraser, “A survey on unit testing practices and problems,” in 2014 IEEE 25th International Symposium on Software Reliability Engineering , 2014, pp. 201–211

2014

Earlier work this paper cites.

N. Tillmann, J. de Halleux, and T. Xie, “Transferring an automated test generation tool to practice: from pex to fakes and code digger,” in ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014 , I. Crnkovic, M. Chechik, and P. Grünbacher, Eds. ACM, 2014, pp. 385–396. [Online]. Available: https://doi.org/10.1145/2642937.2642941

2014

Earlier work this paper cites.

R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” in International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014 , C. S. Pasareanu and D. Marinov, Eds. ACM, 2014, pp. 437–440. [Online]. Available: https://doi.org/10.1145/2610384.2628055

2014

Earlier work this paper cites.

G. Li, E. Andreasen, and I. Ghosh, “Symjs: automatic symbolic testing of JavaScript web applications,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014 , S. Cheung, A. Orso, and M. D. Storey, Eds. ACM, 2014, pp. 449–459. [Online]. Available: https://doi.org/10.1145/2635868.2635913

2014

Earlier work this paper cites.

H. Tanida, T. Uehara, G. Li, and I. Ghosh, “Automatic Unit Test Generation and Execution for JavaScript Program through Symbolic Execution,” in Proceedings of the Ninth International Conference on Software Engineering Advances , 2014, pp. 259–265

2014

Earlier work this paper cites.

A. M. Fard, M. MirzaAghaei, and A. Mesbah, “Leveraging existing tests in automated test generation for web applications,” in ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014 , I. Crnkovic, M. Chechik, and P. Grünbacher, Eds. ACM, 2014, pp. 67–78. [Online]. Available: https://doi.org/10.1145/2642937.2642991

2014

Earlier work this paper cites.

E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling readability to improve unit tests,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015 , E. D. Nitto, M. Harman, and P. Heymans, Eds. ACM, 2015, pp. 107–118. [Online]. Available: https://doi.org/10.1145/2786805.2786838

2015

Cited alongside, same era.

A. M. Fard, A. Mesbah, and E. Wohlstadter, “Generating fixtures for JavaScript unit testing,” in 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015 , M. B. Cohen, L. Grunske, and M. Whalen, Eds. IEEE Computer Society, 2015, pp. 190–200. [Online]. Available: https://doi.org/10.1109/ASE.2015.26

2015

Cited alongside, same era.

——, “JSEFT: automated Javascript unit test generation,” in 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13-17, 2015 . IEEE Computer Society, 2015, pp. 1–10. [Online]. Available: https://doi.org/10.1109/ICST.2015.7102595

2015

Cited alongside, same era.

S. Kim, J. Zhao, Y. Tian, and S. Chandra, “Code prediction by feeding trees to transformers,” in 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021 . IEEE, 2021, pp. 150–162. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00026

2021

Later among the works it cites.

M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, “Self-supervised bug detection and repair,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 27 865–27 876. [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/ea96efc03b9a050d895110db8c4af057-Abstract.html

2021

Later among the works it cites.

L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” 2021. [Online]. Available: https://arxiv.org/abs/2102.07350

Original

2021

Later among the works it cites.