On the Evaluation of Large Language Models in Unit Test Generation

PIE: A Dynamic Failure-Based Technique

Jeffrey M. Voas. 1992 · 1992

Earlier work this paper cites.

Individual comparisons by ranking methods

Frank Wilcoxon. 1992 · 1992

Earlier work this paper cites.

Using Model Checking to Generate Tests from Requirements Specifications. In ESEC / SIGSOFT FSE (Lecture Notes in Computer Science, Vol. 1687) . Springer, 146–162

Angelo Gargantini and Constance L. Heitmeyer. 1999 · 1999

Earlier work this paper cites.

Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In TACAS (Lecture Notes in Computer Science, Vol. 3440) . Springer, 365–381

Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005 · 2005

Earlier work this paper cites.

Combining unit-level symbolic execution and system-level concrete execution for testing NASA software. In ISSTA . ACM, 15–26

Corina S. Pasareanu, Peter C. Mehlitz, David H. Bushnell, Karen Gundy-Burlet, Michael R. Lowry, Suzette Person, and Mark Pape. 2008 · 2008

Earlier work this paper cites.

EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT FSE . ACM, 416–419

Gordon Fraser and Andrea Arcuri. 2011 · 2011

Earlier work this paper cites.

Statistical power analysis for the behavioral sciences

Jacob Cohen. 2013 · 2013

Earlier work this paper cites.

Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In ISSTA . ACM, 437–440

René Just, Darioush Jalali, and Michael D. Ernst. 2014 · 2014

Earlier work this paper cites.

Automated test generation using model checking: an industrial evaluation

Eduard Paul Enoiu, Adnan Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, and Paul Pettersson. 2016 · 2016

Earlier work this paper cites.

The impacts of test automation on software’s cost, quality and time to market

Divya Kumar and Krishn Kumar Mishra. 2016 · 2016

Earlier work this paper cites.

On the diffusion of test smells in automatically generated test code: An empirical study. In 2016 IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST)

Fabio Palomba, Dario Di Nucci, Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2016 · 2016

Earlier work this paper cites.

Shaping program repair space with existing patches and similar code. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (Amsterdam, Netherlands) (ISSTA 2018) . Association for Computing Machinery, New York, NY, USA, 298–309

Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. 2018 · 2018

Earlier work this paper cites.

Context-aware patch generation for better automated program repair. In Proceedings of the 40th international conference on software engineering . 1–11

Ming Wen, Junjie Chen, Rongxin Wu, Dan Hao, and Shing-Chi Cheung. 2018 · 2018

Earlier work this paper cites.

Combining spectrum-based fault localization and statistical debugging: An empirical study. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 502–514

Jiajun Jiang, Ran Wang, Yingfei Xiong, Xiangping Chen, and Lu Zhang. 2019a · 2019

Earlier work this paper cites.

A manual inspection of defects4j bugs and its implications for automatic program repair

Jiajun Jiang, Yingfei Xiong, and Xin Xia. 2019b · 2019

Earlier work this paper cites.

CTRL: A Conditional Transformer Language Model for Controllable Generation

Original

Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019 · 2019

Earlier work this paper cites.

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Original

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 2019

Earlier work this paper cites.

Tbar: Revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis . 31–42

Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F Bissyandé. 2019 · 2019

Earlier work this paper cites.

On the effectiveness of manual and automatic unit test generation: ten years later. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) . IEEE, 121–125

Domenico Serra, Giovanni Grano, Fabio Palomba, Filomena Ferrucci, Harald C Gall, and Alberto Bacchelli. 2019 · 2019

Earlier work this paper cites.

Fine-Tuning Language Models from Human Preferences

Original

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019 · 2019

Earlier work this paper cites.

The Curious Case of Neural Text Degeneration. In ICLR . OpenReview.net

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020 · 2020

Earlier work this paper cites.

Dense Passage Retrieval for Open-Domain Question Answering. In EMNLP (1) . Association for Computational Linguistics, 6769–6781

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al · 2020

Earlier work this paper cites.

SPSS survival manual: A step by step guide to data analysis using IBM SPSS

Julie Pallant. 2020 · 2020

Earlier work this paper cites.

Unit Test Case Generation with Transformers

Original

Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020 · 2020

Earlier work this paper cites.

On learning meaningful assert statements for unit test cases. In ICSE . ACM, 1398–1409

Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020 · 2020

Earlier work this paper cites.

Evaluating Large Language Models Trained on Code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, , et al · 2021

Earlier work this paper cites.

The Power of Scale for Parameter-Efficient Prompt Tuning. In EMNLP (1) . Association for Computational Linguistics, 3045–3059

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Earlier work this paper cites.

Interactive patch filtering as debugging aid. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 239–250

Jingjing Liang, Ruyi Ji, Jiajun Jiang, Shurui Zhou, Yiling Lou, Yingfei Xiong, and Gang Huang. 2021 · 2021

Earlier work this paper cites.

Retrieval Augmented Code Generation and Summarization. In EMNLP (Findings) . Association for Computational Linguistics, 2719–2734

Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021

Earlier work this paper cites.

Toga: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering . 2130–2141

Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022 · 2022

Cited alongside, same era.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022 · 2022

Cited alongside, same era.

Practical Program Repair in the Era of Large Pre-trained Language Models

Original

Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022 · 2022

Cited alongside, same era.

Automated Assertion Generation via Information Retrieval and Its Integration with Deep learning. In ICSE . ACM, 163–174

Hao Yu, Yiling Lou, Ke Sun, Dezhi Ran, Tao Xie, Dan Hao, Ying Li, Ge Li, and Qianxiang Wang. 2022 · 2022

Cited alongside, same era.

Automatic chain of thought prompting in large language models

ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation

Original

Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2023 · 2023

Later among the works it cites.

Llama 2: Open Foundation and Fine-Tuned Chat Models

Original

Hugo Touvron, Louis Martin, Kevin Stone, et al · 2023

Later among the works it cites.

Codet5+: Open code large language models for code understanding and generation

Original

Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023a · 2023

Later among the works it cites.

Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. In ESEC/SIGSOFT FSE . ACM, 172–184

Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023 · 2023

Later among the works it cites.