What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020, 33: 1877–1901

1901

Earlier work this paper cites.

Broder A Z. On the resemblance and containment of documents. 1997. 21–29

1997

Earlier work this paper cites.

Fix G. The design of an automated unit test code generation system. 2009. 743–747

2009

Earlier work this paper cites.

Radford A, Narasimhan K, Salimans T, et al. Improving language understanding by generative pre-training. 2018

2018

Earlier work this paper cites.

Zhang Y, Chen Y, Cheung S C, et al. An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, 2018. 129–140

2018

Earlier work this paper cites.

Iyer S, Konstas I, Cheung A, et al. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588, 2018

Original

2018

Earlier work this paper cites.

Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019, 1: 9

2019

Earlier work this paper cites.

Alawad D, Panta M, Zibran M, et al. An empirical study of the relationships between code readability and software complexity. arXiv preprint arXiv:1909.01760, 2019

Original

2019

Earlier work this paper cites.

Islam M J, Nguyen G, Pan R, et al. A comprehensive study on deep learning bug characteristics. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019. 510–520

2019

Earlier work this paper cites.

Svyatkovskiy A, Deng S K, Fu S, et al. Intellicode compose: Code generation using transformer. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020. 1433–1443

2020

Earlier work this paper cites.

Feng Z, Guo D, Tang D, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020

Original

2020

Earlier work this paper cites.

Gao L, Biderman S, Black S, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

Original

2020

Earlier work this paper cites.

Garcia J, Feng Y, Shen J, et al. A comprehensive study of autonomous vehicle bugs. 2020

2020

Earlier work this paper cites.

Shin J, Nam J. A survey of automatic code generation from natural language. Journal of Information Processing Systems, 2021, 17: 537–555

2021

Earlier work this paper cites.

Wang Y, Wang W, Joty S, et al. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021

Original

2021

Earlier work this paper cites.

Lu S, Guo D, Ren S, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021

Original

2021

Earlier work this paper cites.

Hendrycks D, Basart S, Kadavath S, et al. Measuring coding challenge competence with APPS. In: J Vanschoren, S Yeung, eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html

2021

Earlier work this paper cites.

Austin J, Odena A, Nye M, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

Original

2021

Earlier work this paper cites.

Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

Original

2021

Earlier work this paper cites.

Le H, Wang Y, Gotmare A D, et al. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 2022, 35: 21314–21328

2022

Earlier work this paper cites.

Dehaerne E, Dey B, Halder S, et al. Code generation using machine learning: A systematic review. Ieee Access, 2022, 10: 82434–82455

2022

Earlier work this paper cites.

Zan D, Chen B, Zhang F, et al. Large language models meet nl2code: A survey. arXiv preprint arXiv:2212.09420, 2022

Original

2022

Earlier work this paper cites.

Wang Z, Zhou S, Fried D, et al. Execution-based evaluation for open-domain code generation. arXiv preprint arXiv:2212.10481, 2022

Original

2022

Earlier work this paper cites.

Zhou D, Sch¨arli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022

Original

2022

Earlier work this paper cites.

Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

Original

2022

Earlier work this paper cites.

Wang Y, Kordi Y, Mishra S, et al. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022

Original

2022

Earlier work this paper cites.

Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022, 35: 24824–24837

2022

Earlier work this paper cites.

Dong Q, Li L, Dai D, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

Original

2022

Earlier work this paper cites.

Saunders W, Yeh C, Wu J, et al. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022

Original

2022

Earlier work this paper cites.

Wang G, Wang Z, Chen J, et al. An empirical study on numerical bugs in deep learning programs. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022. 1–5

2022

Earlier work this paper cites.

Xia C S, Zhang L. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022. 959–971

2022

Earlier work this paper cites.

Zhang Z, Chen C, Liu B, et al. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:2311.07989, 2023

Original

2023

Earlier work this paper cites.

Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

Original

2023

Earlier work this paper cites.

Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

Original

2023

Earlier work this paper cites.

Yu Z, Tao Y, Chen L, et al. Beta-coder: On value-based deep reinforcement learning for program synthesis. In: The Twelfth International Conference on Learning Representations, 2023

2023

Cited alongside, same era.

Wang S, Lin B, Sun Z, et al. Two birds with one stone: Boosting code generation and code search via a generative adversarial network. Proceedings of the ACM on Programming Languages, 2023, 7: 486–515

2023

Cited alongside, same era.

Yuan Z, Lou Y, Liu M, et al. No more manual tests? evaluating and improving chatgpt for unit test generation. ArXiv, 2023, abs/2305.04207. URL https://api.semanticscholar.org/CorpusID:258557344

Original

2023

Cited alongside, same era.

Tashtoush Y, Abu-El-Rub N, Darwish O, et al. A notional understanding of the relationship between code readability and software complexity. Information, 2023, 14: 81

2023

Cited alongside, same era.

Bairi R, Sonwane A, Kanade A, et al. Codeplan: Repository-level coding using llms and planning. In: NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

2023

Later among the works it cites.

Weyssow M, Zhou X, Kim K, et al. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv preprint arXiv:2308.10462, 2023

Original

2023

Later among the works it cites.

Shojaee P, Jain A, Tipirneni S, et al. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816, 2023

Original

2023

Later among the works it cites.

Zheng R, Dou S, Gao S, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023

Original

2023

Later among the works it cites.

Liu T, Xu C, McAuley J. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023