TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?

K. Beck, Test driven development: By example . Addison-Wesley Professional, 2002

2002

Earlier work this paper cites.

T. F. Bissyandé, D. Lo, L. Jiang, L. Réveillere, J. Klein, and Y. Le Traon, “Got issues? who cares about it? a large scale investigation of issue trackers from github,” in 2013 IEEE 24th international symposium on software reliability engineering (ISSRE) . IEEE, 2013, pp. 188–197

2013

Earlier work this paper cites.

R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis , 2014, pp. 437–440

2014

Earlier work this paper cites.

M. Martinez, T. Durieux, R. Sommerard, J. Xuan, and M. Monperrus, “Automatic repair of real bugs in java: A large-scale experiment on the defects4j dataset,” Empirical Software Engineering , vol. 22, pp. 1936–1964, 2017

2017

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Earlier work this paper cites.

X. Tan, M. Zhou, and Z. Sun, “A first look at good first issues on github,” in ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 398–409

2020

Earlier work this paper cites.

J. Zhou, S. Wang, C.-P. Bezemer, Y. Zou, and A. E. Hassan, “Studying the association between bountysource bounties and the issue-addressing likelihood of github issue reports,” IEEE Transactions on Software Engineering , vol. 47, no. 12, pp. 2919–2933, 2020

2020

Earlier work this paper cites.

M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” preprint arXiv:2009.05617 , 2020

Original

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, Edwards et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang, “Avatar: A parallel corpus for java-python program translation,” arXiv preprint arXiv:2108.11590 , 2021

Original

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

V. J. Hellendoorn, J. Tsay, M. Mukherjee, and M. Hirzel, “Towards automating code review at scale,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 1479–1482. [Online]. Available: https://doi.org/10.1145/3468264.3473134

2021

Earlier work this paper cites.

M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in github issue and pr comments,” Journal of Systems and Software , vol. 175, p. 110911, 2021

2021

Earlier work this paper cites.

P. Bareiß, B. Souza, M. d’Amorim, and M. Pradel, “Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code,” arXiv preprint arXiv:2206.01335 , 2022

Original

2022

Earlier work this paper cites.

T. Ahmed and P. Devanbu, “Few-shot training LLMs for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–5

2022

Earlier work this paper cites.

Q.-C. Bui, R. Scandariato, and N. E. D. Ferreyra, “Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 464–468

2022

Earlier work this paper cites.

J. Wu, H. He, W. Xiao, K. Gao, and M. Zhou, “Demystifying software release note issues on github,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension , 2022, pp. 602–613

2022

Earlier work this paper cites.

S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring LLM-based general bug reproduction,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2312–2323

2023

Earlier work this paper cites.

B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” arXiv preprint arXiv:2304.03277 , 2023

Original

2023

Cited alongside, same era.

S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu et al. , “Instruction tuning for large language models: A survey,” arXiv preprint arXiv:2308.10792 , 2023

Original

2023

Cited alongside, same era.

N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-based prompt selection for code-related few-shot learning,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2450–2462

2023

Cited alongside, same era.

V. Vikram, C. Lemieux, and R. Padhye, “Can large language models write good property-based tests?” arXiv preprint arXiv:2307.04346 , 2023

Original

2023

Cited alongside, same era.

J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan et al. , “Swe-bench multimodal: Do ai systems generalize to visual software domains?” arXiv preprint arXiv:2410.03859 , 2024

Original

2024

Closest in time.

R. Aleithan, H. Xue, M. M. Mohajer, E. Nnorom, G. Uddin, and S. Wang, “Swe-bench+: Enhanced coding benchmark for LLMs,” arXiv preprint arXiv:2410.06992 , 2024

Original

2024

Closest in time.

D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev et al. , “Coder: Issue resolving with multi-agent and task graphs,” arXiv preprint arXiv:2406.01304 , 2024

Original

2024

Closest in time.

W. Tao, Y. Zhou, W. Zhang, and Y. Cheng, “Magis: LLM-based multi-agent framework for GitHub issue resolution,” arXiv preprint arXiv:2403.17927 , 2024

Original

2024

Closest in time.