HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation

A parallel corpus of python functions and documentation strings for automated code documentation and code generation

Original

Antonio Valerio Miceli Barone and Rico Sennrich. 2017 · 2017

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021 · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Codesc: A large code-description parallel dataset

Original

Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021 · 2021

Earlier work this paper cites.

Multi-lingual evaluation of code generation models

Original

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. 2022 · 2022

Earlier work this paper cites.

Fixeval: Execution-based evaluation of program fixes for competitive programming problems

Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Practical program repair in the era of large pre-trained language models

Original

Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022 · 2022

Earlier work this paper cites.

Xlcost: A benchmark dataset for cross-lingual code intelligence

Original

Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. 2022 · 2022

Earlier work this paper cites.

Impact of code language models on automated program repair

Original

Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023 · 2023

Earlier work this paper cites.

Swe-bench: Can language models resolve real-world github issues?

Original

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023 · 2023

Earlier work this paper cites.

Inferfix: End-to-end program repair with llms

Original

Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023 · 2023

Earlier work this paper cites.

Starcoder: may the source be with you!

Original

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023 · 2023

Earlier work this paper cites.

Repobench: Benchmarking repository-level code auto-completion systems

Original

Tianyang Liu, Canwen Xu, and Julian McAuley. 2023 · 2023

Cited alongside, same era.

Wizardcoder: Empowering code large language models with evol-instruct

Original

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023 · 2023

Cited alongside, same era.

Code llama: Open foundation models for code

Original

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023 · 2023

Cited alongside, same era.

Repository-level prompt generation for large language models of code

Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023 · 2023

Cited alongside, same era.

A survey on language models for code

Livecodebench: Holistic and contamination free evaluation of large language models for code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024 · 2024

Closest in time.

Evocodebench: An evolving code generation benchmark with domain-specific evaluations

Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024 · 2024

Closest in time.

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024 · 2024

Closest in time.

Codestral

Mistral. 2024 · 2024

Closest in time.

Octopack: Instruction tuning code large language models

Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2024 · 2024

Closest in time.