Testing LLMs on Code Generation with Varying Levels of Prompt Specificity

Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language, 2022

Paul Denny, Viraj Kumar, and Nasser Giacaman · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022

Earlier work this paper cites.

Test driven development: A review, 07 2022

Muhammad Usama Pervez and Laiba Eman · 2022

Earlier work this paper cites.

Prompt engineering for chatgpt: A quick guide to techniques, tips, and best practices, 04 2023

Sabit Ekin · 2023

Earlier work this paper cites.

Morgan stanley tmt conference

Scott Guthrie · 2023

Earlier work this paper cites.

Gpt-4 vs. gpt-3.5: A concise showdown, 03 2023

Anis Koubaa · 2023

Cited alongside, same era.

DS-1000: A natural and reliable benchmark for data science code generation

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu · 2023

Cited alongside, same era.

Evaluating the performance of code generation models for solving parsons problems with small prompt variations

Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen · 2023

Cited alongside, same era.

Extending the frontier of chatgpt: Code generation and debugging, 2023

Fardin Ahsan Sakib, Saadat Hasan Khan, and A. H. M. Rezaul Karim · 2023

Cited alongside, same era.

Prompting gpt-3 to be reliable, 2023

Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang · 2023

Closest in time.

Chain-of-thought prompting elicits reasoning in large language models, 2023

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou · 2023

Closest in time.

Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design, 2023

Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt · 2023

Closest in time.

Large language models meet NL2Code: A survey

Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou · 2023

Closest in time.