No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 1877–1901

1901

Earlier work this paper cites.

T. J. McCabe, “A complexity measure,” IEEE Transactions on software Engineering , no. 4, pp. 308–320, 1976

1976

Earlier work this paper cites.

S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

1979

Earlier work this paper cites.

T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean, “Large language models in machine translation,” 2007

2007

Earlier work this paper cites.

S. Haefliger, G. Von Krogh, and S. Spaeth, “Code reuse in open source software,” Management science , vol. 54, no. 1, pp. 180–193, 2008

2008

Earlier work this paper cites.

M. P. Fay and M. A. Proschan, “Wilcoxon-mann-whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules,” Statistics surveys , vol. 4, p. 1, 2010

2010

Earlier work this paper cites.

G. Macbeth, E. Razumiejczyk, and R. D. Ledesma, “Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,” Universitas Psychologica , vol. 10, no. 2, pp. 545–555, 2011

2011

Earlier work this paper cites.

S. Gulwani, “Automating string processing in spreadsheets using input-output examples,” ACM Sigplan Notices , vol. 46, no. 1, pp. 317–330, 2011

2011

Earlier work this paper cites.

W. Stallings, L. Brown, M. D. Bauer, and M. Howard, Computer security: principles and practice . Pearson Upper Saddle River, 2012, vol. 3

2012

Earlier work this paper cites.

T. Gvero, V. Kuncak, I. Kuraj, and R. Piskac, “Complete completion using types and weights,” in Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation , 2013, pp. 27–38

2013

Earlier work this paper cites.

K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078 , 2014

Original

2014

Earlier work this paper cites.

P. Bielik, V. Raychev, and M. Vechev, “Phog: probabilistic model for code,” in International conference on machine learning . PMLR, 2016, pp. 2933–2942

2016

Earlier work this paper cites.

M. Rabinovich, M. Stern, and D. Klein, “Abstract syntax networks for code generation and semantic parsing,” arXiv preprint arXiv:1704.07535 , 2017

Original

2017

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

2017

Earlier work this paper cites.

S. Gulwani, O. Polozov, R. Singh et al. , “Program synthesis,” Foundations and Trends® in Programming Languages , vol. 4, no. 1-2, pp. 1–119, 2017

2017

Earlier work this paper cites.

S. H. Tan, J. Yi, S. Mechtaev, A. Roychoudhury et al. , “Codeflaws: a programming competition benchmark for evaluating automated program repair tools,” in 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) . IEEE, 2017, pp. 180–182

2017

Earlier work this paper cites.

S. Scalabrino, G. Bavota, C. Vendome, M. Linares-Vásquez, D. Poshyvanyk, and R. Oliveto, “Automatically assessing code understandability: How far are we?” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2017, pp. 417–427

2017

Earlier work this paper cites.

J. Li, Y. Wang, M. R. Lyu, and I. King, “Code completion with neural attention and pointer networks,” arXiv preprint arXiv:1711.09573 , 2017

Original

2017

Earlier work this paper cites.

M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Computing Surveys (CSUR) , vol. 51, no. 4, pp. 1–37, 2018

2018

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018

2018

Earlier work this paper cites.

M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations. arxiv 2018,” arXiv preprint arXiv:1802.05365 , vol. 12, 2018

Original

2018

Earlier work this paper cites.

A. Abate, C. David, P. Kesseli, D. Kroening, and E. Polgreen, “Counterexample guided inductive synthesis modulo theories,” in International Conference on Computer Aided Verification . Springer, 2018, pp. 270–288

2018

Earlier work this paper cites.

K. Ashwin, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani, “Neural-guided deductive search for real-time program synthesis from examples,” International Conference on Learning Representations,International Conference on Learning Representations , Feb 2018

2018

Earlier work this paper cites.

U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” Proceedings of the ACM on Programming Languages , vol. 3, no. POPL, pp. 1–29, 2019

2019

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu, “A novel neural source code representation based on abstract syntax tree,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 783–794

2019

Earlier work this paper cites.

Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942 , 2019

Original

2019

Earlier work this paper cites.

W. Ye, R. Xie, J. Zhang, T. Hu, X. Wang, and S. Zhang, “Leveraging code generation to improve code retrieval and summarization via dual learning,” in Proceedings of The Web Conference 2020 , 2020, pp. 2309–2319

2020

Earlier work this paper cites.

M. Tufano, D. Drain, A. Svyatkovskiy, S. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context.” arXiv: Software Engineering,arXiv: Software Engineering , Sep 2020

2020

Earlier work this paper cites.

A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 1433–1443

2020

Cited alongside, same era.

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

Original

2020

Cited alongside, same era.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research , vol. 21, no. 1, pp. 5485–5551, 2020

2020

Cited alongside, same era.

M. Rigaki and S. Garcia, “A survey of privacy attacks in machine learning,” arXiv preprint arXiv:2007.07646 , 2020

Original

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

Original

2022

Later among the works it cites.

B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” arXiv preprint arXiv:2207.10397 , 2022

Original

2022

Later among the works it cites.

Github, “Github copilot · your ai pair programmer,” 2023, https://github.com/features/copilot

2023

Closest in time.

OpenAI, “Chatgpt: Optimizing language models for dialogue,” 2023, https://openai.com/blog/chatgpt/

2023

Closest in time.

Github, “What is ai code generation?” 2023, https://resources.github.com/artificial-intelligence/what-is-ai-code-generation/

2023