CodeJudge: Evaluating Code Generation with Large Language Models

Statistical Power Analysis for the Behavioral Sciences

Jacob Cohen. 1988 · 1988

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Identifying and correcting java programming errors for introductory computer science students

Maria Hristova, Ananya Misra, Megan Rutter, and Rebecca Mercuri. 2003 · 2003

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

Finding and preventing run-time error handling mistakes

Westley Weimer and George C. Necula. 2004 · 2004

Earlier work this paper cites.

Codebleu: a method for automatic evaluation of code synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009

Earlier work this paper cites.

Thinking, fast and slow

Daniel Kahneman. 2011 · 2011

Earlier work this paper cites.

Deadspy: a tool to pinpoint program inefficiencies

Milind Chabbi and John Mellor-Crummey. 2012 · 2012

Earlier work this paper cites.

Meteor universal: Language specific translation evaluation for any target language

Michael Denkowski and Alon Lavie. 2014 · 2014

Earlier work this paper cites.

chrF: character n-gram F-score for automatic MT evaluation

Maja Popović. 2015 · 2015

Earlier work this paper cites.

An approach to identifying error patterns for infrastructure as code

Wei Chen, Guoquan Wu, and Jun Wei. 2018 · 2018

Earlier work this paper cites.

Learning to mine aligned code and natural language pairs from stack overflow

Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018

Earlier work this paper cites.

Spoc: Search-based pseudocode to code

Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019 · 2019

Cited alongside, same era.

Does bleu score work for code migration?

N. Tran, H. Tran, S. Nguyen, H. Nguyen, and T. Nguyen. 2019 · 2019

Cited alongside, same era.

CodeBERT: A pre-trained model for programming and natural languages

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020 · 2020

Cited alongside, same era.

Bertscore: Evaluating text generation with bert

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 · 2020

Cited alongside, same era.

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021

Gptscore: Evaluate as you desire

Original

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023 · 2023

Later among the works it cites.

G-eval: NLG evaluation using gpt-4 with better human alignment

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b · 2023

Later among the works it cites.

Code llama: Open foundation models for code

Original

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023 · 2023

Later among the works it cites.

Pangu-coder2: Boosting large language models for code with ranking feedback

Original

Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang. 2023 · 2023

Later among the works it cites.