Measuring the Influence of Incorrect Code on Test Generation

An experience report on applying software testing academic results in industry: we need usable automated test generation

Andrea Arcuri. 2018 · 1981

Earlier work this paper cites.

Reflective learning: Key to learning from experience

Evelyn M Boyd and Ann W Fales. 1983 · 1983

Earlier work this paper cites.

Learning from errors

Janet Metcalfe. 2017 · 2017

Earlier work this paper cites.

Oracle-guided Program Selection from Large Language Models

Zhiyu Fan, Haifeng Ruan, Sergey Mechtaev, and Abhik Roychoudhury. 2018 · 2018

Earlier work this paper cites.

Can Neural Machine Translation be Improved with User Feedback?. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 3 (Industry Papers) , Srinivas Bangalore, Jennifer Chu-Carroll, and Yunyao Li (Eds.). Association for Computational Linguistics, 92–105

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018 · 2018

Earlier work this paper cites.

Mutation testing advances: an analysis and survey

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019 · 2019

Earlier work this paper cites.

Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.)

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, and Dario Amodei et.al. 2020 · 2020

Earlier work this paper cites.

Codebleu: a method for automatic evaluation of code synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2020

Earlier work this paper cites.

Unit test case generation with transformers and focal context

Original

Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020 · 2020

Earlier work this paper cites.

Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering . 1556–1560

Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, et al · 2020

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al · 2021

Earlier work this paper cites.

Program Synthesis with Large Language Models

Original

Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021a · 2021

Earlier work this paper cites.

Evaluating Large Language Models Trained on Code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, and Wojciech Zaremba et.al. 2021 · 2021

Earlier work this paper cites.

CoDesc: A Large Code-Description Parallel Dataset. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021 (Findings of ACL, Vol. ACL/IJCNLP 2021) , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 210–218

Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md. Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021 · 2021

Earlier work this paper cites.

Measuring Coding Challenge Competence With APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , Joaquin Vanschoren and Sai-Kit Yeung (Eds.)

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021a · 2021

Earlier work this paper cites.

Measuring Coding Challenge Competence With APPS

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021b · 2021

Earlier work this paper cites.

CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 8696–8708

Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021 · 2021

Earlier work this paper cites.

Few-shot training LLMs for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022 . ACM, 177:1–177:5

Toufique Ahmed and Premkumar T. Devanbu. 2022 · 2022

Earlier work this paper cites.

Codet: Code generation with generated tests

Original

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022 · 2022

Earlier work this paper cites.

FixEval: Execution-based Evaluation of Program Fixes for Competitive Programming Problems

Original

Md. Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2022 · 2022

Earlier work this paper cites.

Interactive code generation via test-driven user-intent formalization

Original

Shuvendu K Lahiri, Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madanlal Musuvathi, Piali Choudhury, Curtis von Veh, Jeevana Priya Inala, Chenglong Wang, et al · 2022

Earlier work this paper cites.

Competition-Level Code Generation with AlphaCode

Original

Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022 · 2022

Earlier work this paper cites.

Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . ACM, 2241–2252

Amir M. Mir, Evaldas Latoskinas, Sebastian Proksch, and Georgios Gousios. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.)

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

SantaCoder: don’t reach for the stars!

Original

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Ferrandis, Niklas Muennighoff, Mayank Mishra, and Leandro von Werra et.al. 2023 · 2023

Earlier work this paper cites.

Teaching Large Language Models to Self-Debug

Original

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023 · 2023

Cited alongside, same era.

DeepSeek Coder: Let the Code Write Itself

DeepSeekAI. 2023 · 2023

Cited alongside, same era.

Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT

Original

Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023b · 2023

Cited alongside, same era.

InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023 · 2023

Cited alongside, same era.

Kernelgpt: Enhanced kernel fuzzing via large language models

Original

Chenyuan Yang, Zijie Zhao, and Lingming Zhang. 2023b · 2023

Later among the works it cites.

ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.)

Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023 · 2023

Later among the works it cites.

Language agent tree search unifies reasoning acting and planning in language models

Original

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023 · 2023

Later among the works it cites.

MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

Jianbo Dai, Jianqiao Lu, Yunlong Feng, Rongju Ruan, Ming Cheng, Haochen Tan, and Zhijiang Guo. 2024 · 2024