HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

Using benchmarking to advance research: A challenge to software engineering. In 25th International Conference on Software Engineering, 2003. Proceedings. IEEE, 74–83

Susan Elliott Sim, Steve Easterbrook, and Richard C Holt. 2003 · 2003

Earlier work this paper cites.

The education of a software engineer. In Proceedings. 19th International Conference on Automated Software Engineering, 2004. IEEE, xviii–xxvii

Mehdi Jazayeri. 2004 · 2004

Earlier work this paper cites.

Communication and co-ordination practices in software engineering projects

Ian R McChesney and Seamus Gallagher. 2004 · 2004

Earlier work this paper cites.

Software engineering: a practitioner’s approach

Roger S Pressman. 2005 · 2005

Earlier work this paper cites.

Collaboration in software engineering: A roadmap. In Future of Software Engineering (FOSE’07) . IEEE, 214–225

Jim Whitehead. 2007 · 2007

Earlier work this paper cites.

Collaborative software engineering: challenges and prospects

Ivan Mistrík, John Grundy, Andre Van der Hoek, and Jim Whitehead. 2010 · 2010

Earlier work this paper cites.

Applications of ontologies in requirements engineering: a systematic review of the literature

Diego Dermeval, Jéssyka Vilela, Ig Ibert Bittencourt, Jaelson Castro, Seiji Isotani, Patrick Brito, and Alan Silva. 2016 · 2016

Earlier work this paper cites.

Abstract Syntax Networks for Code Generation and Semantic Parsing

Original

M. Rabinovich, M. Stern, and D. Klein. 2017 · 2017

Earlier work this paper cites.

Attention is All You Need. In Advances in Neural Information Processing Systems , Vol. 30

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017 · 2017

Earlier work this paper cites.

Can Neural Machine Translation Be Improved with User Feedback?. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018 · 2018

Earlier work this paper cites.

Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th international conference on mining software repositories . 476–486

Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018

Earlier work this paper cites.

Code2Vec: Learning Distributed Representations of Code

U. Alon, M. Zilberstein, O. Levy, and E. Yahav. 2019 · 2019

Earlier work this paper cites.

CodeBERT: A Pre-trained Model for Programming and Natural Languages

Original

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, and D. Jiang et al. 2020 · 2020

Earlier work this paper cites.

Intellicode Compose: Code Generation Using Transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1433–1443

A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan. 2020 · 2020

Earlier work this paper cites.

Unit Test Case Generation with Transformers and Focal Context

M. Tufano, D. Drain, A. Svyatkovskiy, S. Deng, and N. Sundaresan. 2020 · 2020

Earlier work this paper cites.

Leveraging Code Generation to Improve Code Retrieval and Summarization via Dual Learning. In Proceedings of The Web Conference 2020 . 2309–2319

W. Ye, R. Xie, J. Zhang, T. Hu, X. Wang, and S. Zhang. 2020 · 2020

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al · 2021

Earlier work this paper cites.

InferCode: Self-supervised Learning of Code Representations by Predicting Subtrees. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . 1186–1197

N. D. Bui, Y. Yu, and L. Jiang. 2021 · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021

Earlier work this paper cites.

Measuring coding challenge competence with apps

Original

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al · 2021

Earlier work this paper cites.

Requirement engineering challenges: A systematic mapping study on the academic and the industrial perspective

Muhammad Tukur, Sani Umar, and Jameleddine Hassine. 2021 · 2021

Earlier work this paper cites.

GPT-J-6B: A 6 billion parameter autoregressive language model

Ben Wang and Aran Komatsuzaki. 2021 · 2021

Earlier work this paper cites.

CodeT5: Identifier-Aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Original

Y. Wang, W. Wang, S. Joty, and S. C. Hoi. 2021 · 2021

Earlier work this paper cites.

Constitutional AI: Harmlessness from AI Feedback

Original

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022 · 2022

Earlier work this paper cites.

Improving Alignment of Dialogue Agents via Targeted Human Judgements

Original

Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin J. Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sona Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022 · 2022

Earlier work this paper cites.

Large Language Models Can Self-Improve

Original

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022 · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al · 2022

Earlier work this paper cites.

An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories . 1–5

N. Nguyen and S. Nadi. 2022 · 2022

Earlier work this paper cites.

Codegen: An open large language model for code with multi-turn program synthesis

Original

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022 · 2022

Earlier work this paper cites.

Training Language Models to Follow Instructions with Human Feedback. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS)

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

What is it like to program with artificial intelligence?

Original

Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022 · 2022

Cited alongside, same era.

An Empirical Study of Code Smells in Transformer-Based Code Generation Techniques. In 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) . 71–82

M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. Santos. 2022 · 2022

Cited alongside, same era.

Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. In Proceedings of the Genetic and Evolutionary Computation Conference . 1019–1027

D. Sobania, M. Briesch, and F. Rothlauf. 2022 · 2022

Cited alongside, same era.

Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . 1–7

P. Vaithilingam, T. Zhang, and E. L. Glassman. 2022 · 2022

Toolformer: Language models can teach themselves to use tools

Original

Timo Schick, Jane Dwivedi-Yu, Patrick Schramowski, and Jacob Andreas. 2023 · 2023

Later among the works it cites.

Reflexion: Language agents with verbal reinforcement learning

Original

Noah Shinn, Nelson Labash, Mark Skreta, Shunyu Yao, and Karthik Narasimhan. 2023 · 2023

Later among the works it cites.

Is chatgpt a good nlg evaluator? a preliminary study

Original

Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a · 2023

Later among the works it cites.

A survey on large language model based autonomous agents

Original

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al · 2023

Later among the works it cites.