Large Language Model Critics for Execution-Free Evaluation of Code Changes

Metrics for assessing a software system’s maintainability

Paul W. Oman and Jack R. Hagemeister · 1992

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002

Earlier work this paper cites.

A differencing algorithm for object-oriented programs

Taweesup Apiwattanapong, Alessandro Orso, and Mary Jean Harrold · 2004

Earlier work this paper cites.

Crystalbleu: precisely and efficiently measuring the similarity of code

Aryaz Eghbali and Michael Pradel · 2004

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin · 2004

Earlier work this paper cites.

Chianti: a change impact analysis tool for java programs

Xiaoxia Ren, Barbara G. Ryder, Maximilian Störzer, and Frank Tip · 2005

Earlier work this paper cites.

Does bleu score work for code migration?

Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen · 2019

Earlier work this paper cites.

Codebert: A pre-trained model for programming and natural languages

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou · 2020

Earlier work this paper cites.

Dlfix: context-based code transformation learning for automated program repair

Yi Li, Shaohua Wang, and Tien N. Nguyen · 2020

Earlier work this paper cites.

Codebleu: a method for automatic evaluation of code synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma · 2020

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021

Earlier work this paper cites.

Linevul: A transformer-based line-level vulnerability prediction

Michael Fu and Chakkrit Tantithamthavorn · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Original

Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou · 2022

Cited alongside, same era.

Better patching using LLM prompting, via self-consistency

Toufique Ahmed and Premkumar T. Devanbu · 2023

Cited alongside, same era.

Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang · 2023

Cited alongside, same era.

Codescore: Evaluating code generation by learning code execution

Original

Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin · 2023

Cited alongside, same era.

Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation

Original

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou · 2023

Iterative refinement of project-level code context for precise code generation with compiler feedback

Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, and Xuanhua Shi · 2024

Later among the works it cites.

R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models

Original

Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, Yuanxing Zhang, Wenbo Su, Bangyu Xiang, Tiezheng Ge, and Bo Zheng · 2024

Later among the works it cites.

Code droid: A technical report, 2024

Factory Code Droid · 2024

Later among the works it cites.

Gru.ai ranks first in openai’s latest swe-bench verified evaluation, 2024

Gru.ai · 2024

Later among the works it cites.

Swe-bench: Can language models resolve real-world github issues?

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan · 2024

Later among the works it cites.