CodeArena: A Collective Evaluation Platform for LLM Code Generation

RESTful web services

Leonard Richardson and Sam Ruby. 2008 · 2008

Earlier work this paper cites.

Rest apis: A large-scale analysis of compliance with principles and best practices

Carlos Rodríguez, Marcos Baez, Florian Daniel, Fabio Casati, Juan Carlos Trabucco, Luigi Canali, and Gianraffaele Percannella. 2016 · 2016

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Measuring coding challenge competence with apps

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Fuzzing: a survey for roadmap

Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Purple llama cyberseceval: A secure coding benchmark for language models

Original

Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. 2023 · 2023

Earlier work this paper cites.

Mistral 7b

Original

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 · 2023

Earlier work this paper cites.

Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023 · 2023

Cited alongside, same era.

Code llama: Open foundation models for code

Original

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023 · 2023

Cited alongside, same era.

Gemini: a family of highly capable multimodal models

Original

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023

Cited alongside, same era.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Cited alongside, same era.

Mercury: An efficiency benchmark for llm code synthesis

Original

Mingzhe Du, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. 2024 · 2024

Later among the works it cites.

Effibench: Benchmarking the efficiency of automatically generated code

Original

Dong Huang, Jie M Zhang, Yuhao Qing, and Heming Cui. 2024 · 2024

Later among the works it cites.

Livecodebench: Holistic and contamination free evaluation of large language models for code

Original

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024 · 2024

Later among the works it cites.

Leetcode - the world’s leading online programming learning platform

LeetCode. 2024 · 2024

Later among the works it cites.

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation