DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation

Language models are few-shot learners

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020 · 1901

Earlier work this paper cites.

Program Synthesis with Large Language Models

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; and Michalewski, H. 2021 · 2021

Earlier work this paper cites.

Evaluating Large Language Models Trained on Code

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; and Pinto, H. P. d. O. 2021 · 2021

Earlier work this paper cites.

Measuring Coding Challenge Competence With APPS

Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; and Arora, A. 2021 · 2021

Earlier work this paper cites.

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

Original

Cassano, F.; Gouwar, J.; Nguyen, D.; Nguyen, S.; Phipps-Costin, L.; Pinckney, D.; Yee, M.-H.; Zi, Y.; Anderson, C. J.; Feldman, M. Q.; Guha, A.; Greenberg, M.; and Jangda, A. 2022 · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Li, Y.; Choi, D.; Chung, J.; Kushman, N.; Schrittwieser, J.; Leblond, R.; Eccles, T.; Keeling, J.; Gimeno, F.; Dal Lago, A.; et al. 2022 · 2022

Earlier work this paper cites.

SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques

Siddiq, M. L.; and Santos, J. C. 2022 · 2022

Earlier work this paper cites.

CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation

Zan, D.; Chen, B.; Yang, D.; Lin, Z.; and Kim, M. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023 · 2023

Earlier work this paper cites.

Qwen technical report

Original

Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023 · 2023

Earlier work this paper cites.

ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation

Du, X.; Liu, M.; Wang, K.; Wang, H.; and Liu, J. 2023 · 2023

Cited alongside, same era.

MathPrompter: Mathematical Reasoning using Large Language Models

Original

Imani, S.; Du, L.; and Shrivastava, H. 2023 · 2023

Cited alongside, same era.

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Rogers, A.; Gardner, M.; and Augenstein, I. 2023 · 2023

Cited alongside, same era.

Code llama: Open foundation models for code

Original

Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X. E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al. 2023 · 2023

Cited alongside, same era.

Llama 2: Open foundation and fine-tuned chat models

Original

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023 · 2023

DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence

Original

Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y.; Li, Y.; et al. 2024 · 2024

Closest in time.

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

Li, J.; Li, G.; Zhao, Y.; Li, Y.; and Liu, H. 2024 · 2024

Closest in time.

StarCoder 2 and The Stack v2: The Next Generation

Lozhkov, A.; Li, R.; Allal, L. B.; Cassano, F.; and Lamy-Poirier, J. 2024 · 2024

Closest in time.

Exploring Multi-Lingual Bias of Large Code Models in Code Generation

Original

Wang, C.; Li, Z.; Gao, C.; Wang, W.; Peng, T.; Huang, H.; Deng, Y.; Wang, S.; and Lyu, M. R. 2024 · 2024

Closest in time.

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

Xie, Y.; Xie, A.; Sheth, D.; Liu, P.; and Fried, D. 2024 · 2024

Closest in time.