2022

Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction

Kang, Sungmin, Yoon, Juyeon, Yoo, Shin

Understand

Many automated test generation techniques have been developed to aid developers with writing tests.

To facilitate full automation, most existing techniques aim to either increase coverage, or generate exploratory inputs.
However, existing test generation techniques largely fall short of achieving more semantic objectives, such as generating tests to reproduce a given bug report.
Reproducing bugs is nonetheless important, as our empirical study shows that the number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size.

Built on

W. Miller and D. L. Spooner, “Automatic generation of floating-point test data,” IEEE Transactions on Software Engineering , vol. 2, no. 3, pp. 223–226, 1976
1976
Earlier work this paper cites.
C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in OOPSLA ’07: Companion to the 22nd ACM SIGPLAN conference on Object oriented programming systems and applications companion . ACM, 2007, pp. 815–816
2007
Earlier work this paper cites.
R. Premraj, T. Zimmermann, S. Kim, and N. Bettenburg, “Extracting structural information from bug reports,” in Proceedings of the 2008 international workshop on Mining software repositories - MSR ’08 , ACM Press. ACM Press, 05/2008 2008, pp. 27–30
2008
Earlier work this paper cites.
M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers,” CoRR , vol. abs/2009.05617, 2020. [Online]. Available: https://arxiv.org/abs/2009.05617
Original
2009
Earlier work this paper cites.
G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Trans. Softw. Eng. , vol. 39, no. 2, pp. 276–291, Feb. 2013
2013
Earlier work this paper cites.
R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis , ser. ISSTA 2014. ACM, 2014, pp. 437–440
2014
Earlier work this paper cites.
E. Daka and G. Fraser, “A survey on unit testing practices and problems,” in 2014 IEEE 25th International Symposium on Software Reliability Engineering , 2014, pp. 201–211
2014
Earlier work this paper cites.
N. Chen and S. Kim, “Star: Stack trace based automatic crash reproduction via symbolic execution,” IEEE transactions on software engineering , vol. 41, no. 2, pp. 198–220, 2014
2014
Earlier work this paper cites.
M. Nayrolles, A. Hamou-Lhadj, S. Tahar, and A. Larsson, “Jcharming: A bug reproduction approach using crash traces and directed model checking,” in 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) , 2015, pp. 101–110
2015
Earlier work this paper cites.
T.-D. B. Le, R. J. Oentaryo, and D. Lo, “Information retrieval and spectrum based bug localization: Better together,” in Proceedings of the 10th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE 2015. ACM, 2015, pp. 579–590
2015
Earlier work this paper cites.
E. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering , vol. 41, no. 5, pp. 507–525, May 2015
2015
Earlier work this paper cites.
J. Xuan, X. Xie, and M. Monperrus, “Crash reproduction via test case mutation: Let existing test cases help,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, pp. 910–913
2015
Earlier work this paper cites.
R. Alur, R. Bodík, E. Dallal, D. Fisman, P. Garg, G. Juniwal, H. Kress-Gazit, P. Madhusudan, M. M. K. Martin, M. Raghothaman, S. Saha, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa, “Syntax-guided synthesis,” in Dependable Software Systems Engineering , ser. NATO Science for Peace and Security Series, D: Information and Communication Security, M. Irlbeck, D. A. Peled, and A. Pretschner, Eds. IOS Press, 2015, vol. 40, pp. 1–25. [Online]. Available: https://doi.org/10.3233/978-1-61499-495-4-1
2015
Earlier work this paper cites.
P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in Proceedings of the 25th International Symposium on Software Testing and Analysis , ser. ISSTA 2016. Association for Computing Machinery, 2016, pp. 165–176. [Online]. Available: https://doi.org/10.1145/2931037.2931051
2016
Earlier work this paper cites.

Similar

M. Martinez, T. Durieux, R. Sommerard, J. Xuan, and M. Martin, “Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset,” Empirical Software Engineering , vol. 22, pp. 1936–1964, 2016
2016
Cited alongside, same era.
Y. Xiong, J. Wang, R. Yan, J. Zhang, S. Han, G. Huang, and L. Zhang, “Precise condition synthesis for program repair,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) , 2017, pp. 416–426
2017
Cited alongside, same era.
M. Soltani, P. Derakhshanfar, A. Panichella, X. Devroey, A. Zaidman, and A. van Deursen, “Single-objective versus multi-objectivized optimization for evolutionary crash reproduction,” in Search-Based Software Engineering , T. E. Colanzi and P. McMinn, Eds. Springer International Publishing, 2018, pp. 325–340
2018
Cited alongside, same era.
R. Just, C. Parnin, I. Drosos, and M. D. Ernst, “Comparing developer-provided to user-provided tests for fault localization and automated program repair,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2018, pp. 287–297
2018
Cited alongside, same era.
M. Fazzini, M. Prammer, M. d’Amorim, and A. Orso, “Automatically translating bug reports into test cases for mobile apps,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2018. ACM, 2018, pp. 141–152. [Online]. Available: http://doi.acm.org/10.1145/3213846.3213869
2018
Cited alongside, same era.
A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, M. Monperrus, J. Klein, and Y. Le Traon, “Ifixr: Bug report driven program repair,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2019. Association for Computing Machinery, 2019, pp. 314–325. [Online]. Available: https://doi.org/10.1145/3338906.3338935
2019
Cited alongside, same era.
X. Li, W. Li, Y. Zhang, and L. Zhang, “Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2019. Association for Computing Machinery, 2019, pp. 169–180. [Online]. Available: https://doi.org/10.1145/3293882.3330574
2019
Cited alongside, same era.
P. S. Kochhar, X. Xia, and D. Lo, “Practitioners’ views on good software testing practices,” in Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice , ser. ICSE-SEIP ’19. IEEE Press, 2019, pp. 61–70. [Online]. Available: https://doi.org/10.1109/ICSE-SEIP.2019.00015
2019
Cited alongside, same era.
U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019. [Online]. Available: https://openreview.net/forum?id=H1gKYo09tX
2019
Cited alongside, same era.
M. Motwani and Y. Brun, “Automatically generating precise oracles from structured natural language specifications,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , 2019, pp. 188–199
2019
Cited alongside, same era.
Y. Zhao, T. Yu, T. Su, Y. Liu, W. Zheng, J. Zhang, and W. G.J. Halfond, “Recdroid: Automatically reproducing android application crashes from bug reports,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , 2019, pp. 128–139
2019
Cited alongside, same era.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020
2020
Cited alongside, same era.

Then

M. Soltani, P. Derakhshanfar, X. Devroey, and A. Van Deursen, “A benchmark-based evaluation of search-based crash reproduction,” Empirical Software Engineering , vol. 25, no. 1, pp. 96–138, 2020
2020
Later among the works it cites.
M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” 2020
2020
Later among the works it cites.
P. Derakhshanfar, X. Devroey, A. Panichella, A. Zaidman, and A. van Deursen, “Botsing, a search-based crash reproduction framework for java,” in 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2020, pp. 1278–1282
2020
Later among the works it cites.
R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be optimized? developer survey, optimization guidelines, and case studies,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. Association for Computing Machinery, 2021, pp. 1281–1291. [Online]. Available: https://doi.org/10.1145/3468264.3473922
2021
Later among the works it cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Later among the works it cites.
L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–7
2021
Later among the works it cites.
R. Tinn, H. Cheng, Y. Gu, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, “Fine-tuning large neural language models for biomedical natural language processing,” CoRR , vol. abs/2112.07869, 2021
Original
2021
Later among the works it cites.
L. Wang, H. Hu, L. Sha, C. Xu, K. Wong, and D. Jiang, “Finetuning large-scale pre-trained language models for conversational recommendation with knowledge graph,” CoRR , vol. abs/2110.07477, 2021
Original
2021
Later among the works it cites.
B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” arXiv preprint arXiv:2207.10397 , 2022
Original
2022
Closest in time.
A. Sarkar, A. Gordon, C. Negreanu, C. Poelitz, S. Srinivasa Ragavan, and B. Zorn, “What is it like to program with artificial intelligence?” arXiv preprint arXiv:2208.06213 , 2022
Original
2022
Closest in time.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” 2022. [Online]. Available: https://arxiv.org/abs/2205.11916
Original
2022
Closest in time.
Y. Noller, R. Shariffdeen, X. Gao, and A. Roychoudhury, “Trust enhancement issues in program repair,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. Association for Computing Machinery, 2022, pp. 2228–2240. [Online]. Available: https://doi.org/10.1145/3510003.3510040
2022
Closest in time.
C. Thunes, “javalang: Pure Python Java parser and tools,” https://github.com/c2nes/javalang , 2022
2022
Closest in time.
Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago et al. , “Competition-level code generation with AalphaCode,” arXiv preprint arXiv:2203.07814 , 2022
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…