Fetching the paper…

Reading the bibliography…

2024

SWE-Bench+: Enhanced Coding Benchmark for LLMs

Aleithan, Reem, Xue, Haoran, Mohajer, Mohammad Mahdi et al.

Original
PDF
Post on X

Understand

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding.

To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al.
introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories.
Several impressive LLM-based toolkits recently are developed and evaluated on this dataset.

SWE-Bench+: Enhanced Coding Benchmark for LLMs · Around

Built on

Evaluating large language models trained on code, 2021
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021
Earlier work this paper cites.
Competition-level code generation with alphacode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022
Earlier work this paper cites.
Are we building on the rock? on the importance of data preprocessing for code summarization
Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang · 2022
Earlier work this paper cites.
Less training, more repairing please: Revisiting automated program repair via zero-shot learning, 2022
Original
Chunqiu Steven Xia and Lingming Zhang · 2022
Earlier work this paper cites.
Large language models for software engineering: Survey and open problems, 2023
Original
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang · 2023
Earlier work this paper cites.
Automated program repair in the era of large pre-trained language models
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang · 2023
Earlier work this paper cites.
Large language models meet nl2code: A survey, 2023
Original
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou · 2023
Earlier work this paper cites.
Language models are better bug detector through code-pair classification, 2024
Original
Kamel Alrashedy and Ahmed Binjahlan · 2024
Earlier work this paper cites.

Similar

Coder: Issue resolving with multi-agent and task graphs, 2024
Original
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang · 2024
Cited alongside, same era.
Enhanced automated code vulnerability repair using large language models
David de Fitero-Dominguez, Eva Garcia-Lopez, Antonio Garcia-Cabot, and Jose-Javier Martinez-Herraiz · 2024
Cited alongside, same era.
Evaluating large language models in class-level code generation
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou · 2024
Cited alongside, same era.
A deep dive into large language models for automated bug localization and repair
Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp · 2024
Cited alongside, same era.
Large language models for software engineering: A systematic literature review, 2024

Then

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang · 2024
Closest in time.
Large language model-based code generation for the control of construction assembly robots: A hierarchical generation approach
Hanbin Luo, Jianxin Wu, Jiajing Liu, and Maxwell Fordjour Antwi-Afari · 2024
Closest in time.
Can github issues be solved with tree of thoughts?, 2024
Original
Ricardo La Rosa, Corey Hulse, and Bangdi Liu · 2024
Closest in time.
Magis: Llm-based multi-agent framework for github issue resolution, 2024
Original
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng · 2024
Closest in time.
Agentless: Demystifying llm-based software engineering agents, 2024
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang · 2024

Cited alongside, same era.

A survey on large language models for code generation, 2024

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim · 2024

Cited alongside, same era.

Swe-bench: Can language models resolve real-world github issues?, 2024

Original

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2024

Cited alongside, same era.

Towards efficient data wrangling with llms using code generation

Xue Li and Till Döhmen · 2024

Cited alongside, same era.

Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023a

Original

Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang

Cited in the paper.

Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, 2023b

Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang

Cited in the paper.

Swe-agent: Agent-computer interfaces enable automated software engineering, 2024b

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

Cited in the paper.

Diversity empowers intelligence: Integrating expertise of software engineering agents, 2024a

Original

Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong

Cited in the paper.

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang · 2024

Closest in time.

Cref: An llm-based conversational software repair framework for programming tutors

Boyang Yang, Haoye Tian, Weiguo Pian, Haoran Yu, Haitao Wang, Jacques Klein, Tegawendé F. Bissyandé, and Shunfu Jin · 2024

Closest in time.

Swe-bench-java: A github issue resolving benchmark for java, 2024

Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng, Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, and Qianxiang Wang · 2024

Closest in time.

A survey of large language models for code: Evolution, benchmarking, and future trends, 2024

Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen · 2024

Closest in time.