How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark

On composite numbers p p which satisfy the Fermat congruence a p − 1 ≡ 1 ( mod p ) a^{p-1}\equiv 1\pmod{p}

Robert D. Carmichael · 1912

Earlier work this paper cites.

A class of statistics with asymptotically normal distribution

Wassily Hoeffding · 1948

Earlier work this paper cites.

Estimates of location based on rank tests

Joseph L. Hodges Jr. and Erich L. Lehmann · 1963

Earlier work this paper cites.

Experiments with a heuristic compiler

Herbert A. Simon · 1963

Earlier work this paper cites.

Validity of the single processor approach to achieving large scale computing capabilities

Gene M. Amdahl · 1967

Earlier work this paper cites.

PROW: A step toward automatic program writing

Richard J. Waldinger and Richard CT Lee · 1969

Earlier work this paper cites.

Toward automatic program synthesis

Zohar Manna and Richard J. Waldinger · 1971

Earlier work this paper cites.

Inferring LISP programs from examples

David E. Shaw, William R. Swartout, and C. Cordell Green · 1975

Earlier work this paper cites.

Bootstrap methods: Another look at the jackknife

Bradley Efron · 1979

Earlier work this paper cites.

Application of theorem proving to problem solving

Cordell Green · 1981

Earlier work this paper cites.

Self-adjusting binary search trees

Daniel Dominic Sleator and Robert Endre Tarjan · 1985

Earlier work this paper cites.

Rao-Blackwellisation of sampling schemes

George Casella and Christian P. Robert · 1996

Earlier work this paper cites.

Estimating medical costs with censored data

Heejung Bang and Anastasios A. Tsiatis · 2000

Earlier work this paper cites.

PRIMES is in P

Manindra Agrawal, Neeraj Kayal, and Nitin Saxena · 2004

Earlier work this paper cites.

Towards a statistical methodology to evaluate program speedups and their optimisation techniques

Sid-Ahmed-Ali Touati · 2009

Earlier work this paper cites.

Automating string processing in spreadsheets using input-output examples

Sumit Gulwani · 2011

Earlier work this paper cites.

Program Synthesis , volume 4 of Foundations and Trends® in Programming Languages

Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh · 2017

Earlier work this paper cites.

Neural-guided deductive search for real-time program synthesis from examples

Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani · 2018

Earlier work this paper cites.

Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev · 2018

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020

Cited alongside, same era.

Program synthesis with large language models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton · 2021

Cited alongside, same era.

EE-Net: Exploitation-exploration neural networks in contextual bandits

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He · 2021

Cited alongside, same era.

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021

B-coder: Value-based deep reinforcement learning for program synthesis

Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang · 2023

Later among the works it cites.

CodeGeeX: A pre-trained model for code generation with multilingual evaluations on Humaneval-X

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang · 2023

Later among the works it cites.

The Claude 3 model family: Opus, Sonnet, Haiku, 2024

Anthropic · 2024

Closest in time.

Group fairness via group consensus

Eunice Chan, Zhining Liu, Ruizhong Qiu, Yuheng Zhang, Ross Maciejewski, and Hanghang Tong · 2024

Closest in time.

WAPITI: A watermark for finetuned open-source LLMs

Lingjie Chen, Ruizhong Qiu, Siyu Yuan, Zhining Liu, Tianxin Wei, Hyunsik Yoo, Zhichen Zeng, Deqing Yang, and Hanghang Tong · 2024

Closest in time.

Mercury: An efficiency benchmark for LLM code synthesis