GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging

MLench: benchmarking Machine Learning Services Against Human Experts

Liu, Y.; Zhang, H.; Zeng, L.; Wu, W.; and Zhang, C. 2018 · 2018

Earlier work this paper cites.

Evaluating Large Language Models trained on Code

Original

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021 · 2021

Earlier work this paper cites.

Measuring Coding Challenge Competence with Apps

Original

Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021 · 2021

Earlier work this paper cites.

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Lu, S.; Guo, D.; Ren, S.; Huang, J.; Svyatkovskiy, A.; Blanco, A.; Clement, C.; Drain, D.; Jiang, D.; Tang, D.; et al. 2021 · 2021

Earlier work this paper cites.

Jigsaw: Large Language Models Meet Program Synthesis

Jain, N.; Vaidyanath, S.; Iyer, A.; Natarajan, N.; Parthasarathy, S.; Rajamani, S.; and Sharma, R. 2022 · 2022

Earlier work this paper cites.

Competition-level Code Generation with Alphacode

Li, Y.; Choi, D.; Chung, J.; Kushman, N.; Schrittwieser, J.; Leblond, R.; Eccles, T.; Keeling, J.; Gimeno, F.; Dal Lago, A.; et al. 2022 · 2022

Earlier work this paper cites.

Execution-based Evaluation for Open-domain Code Generation

Original

Wang, Z.; Zhou, S.; Fried, D.; and Neubig, G. 2022 · 2022

Earlier work this paper cites.

Crosscodeeval: A Diverse and Multilingual Benchmark for Cross-file Code Completion

Ding, Y.; Wang, Z.; Ahmad, W.; Ding, H.; Tan, M.; Jain, N.; Ramanathan, M. K.; Nallapati, R.; Bhatia, P.; Roth, D.; et al. 2023 · 2023

Earlier work this paper cites.

Classeval: A Manually-crafted Benchmark for Evaluating LLMs on Class-level Code Generation

Original

Du, X.; Liu, M.; Wang, K.; Wang, H.; Liu, J.; Chen, Y.; Feng, J.; Sha, C.; Peng, X.; and Lou, Y. 2023 · 2023

Earlier work this paper cites.

MLAgentBench: Evaluating language Agents on Machine Learning Experimentation

Original

Huang, Q.; Vora, J.; Liang, P.; and Leskovec, J. 2023 · 2023

Earlier work this paper cites.

Swe-bench: Can Language Models Resolve Real-World GitHub Issues??

Original

Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2023 · 2023

Earlier work this paper cites.

DS-1000: A natural and reliable benchmark for data science code generation

Lai, Y.; Li, C.; Wang, Y.; Zhang, T.; Zhong, R.; Zettlemoyer, L.; Yih, W.-t.; Fried, D.; Wang, S.; and Yu, T. 2023 · 2023

Earlier work this paper cites.

Api-bank: A Comprehensive Benchmark for Tool-augmented LLMs

Original

Li, M.; Zhao, Y.; Yu, B.; Song, F.; Li, H.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023 · 2023

Earlier work this paper cites.

Repobench: Benchmarking Repository-level Code Auto-completion Systems

Original

Liu, T.; Xu, C.; and McAuley, J. 2023 · 2023

Earlier work this paper cites.

Gitagent: Facilitating Autonomous Agent with Github by Tool Extension

Original

Lyu, B.; Cong, X.; Yu, H.; Yang, P.; Qin, Y.; Ye, Y.; Lu, Y.; Zhang, Z.; Yan, Y.; Lin, Y.; et al. 2023 · 2023

Earlier work this paper cites.

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

Original

Tang, X.; Liu, Y.; Cai, Z.; Shao, Y.; Lu, J.; Zhang, Y.; Deng, Z.; Hu, H.; An, K.; Huang, R.; et al. 2023 · 2023

Earlier work this paper cites.

Toolcoder: Teach Code Generation Models to Use Api Search Tools

Original

Zhang, K.; Zhang, H.; Li, G.; Li, J.; Li, Z.; and Jin, Z. 2023 · 2023

Earlier work this paper cites.

Claude 3.5 Sonnet

Anthropic. 2024 · 2024

Cited alongside, same era.

Large language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives

Gao, C.; Lan, X.; Li, N.; Yuan, Y.; Ding, J.; Zhou, Z.; Xu, F.; and Li, Y. 2024 · 2024

Cited alongside, same era.

Self-organized agents: A LLM Multi-agent Framework toward Ultra Large-scale Code Generation and Optimization

Original

Ishibashi, Y.; and Nishimura, Y. 2024 · 2024

Cited alongside, same era.

Livecodebench: Holistic and contamination free evaluation of large language models for code

Original

Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2024 · 2024

Cited alongside, same era.

Devbench: A Comprehensive Benchmark for Software Development

Li, B.; Wu, W.; Tang, Z.; Shi, L.; Yang, J.; Li, J.; Yao, S.; Qian, C.; Hui, B.; Zhang, Q.; et al. 2024 · 2024

Cited alongside, same era.

xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

Original

Chen, K.; Ren, Y.; Liu, Y.; Hu, X.; Tian, H.; Xie, T.; Liu, F.; Zhang, H.; Liu, H.; Gong, Y.; et al. 2025 · 2025

Closest in time.

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

Dong, P.; Tang, Z.; Liu, X.; Li, L.; Chu, X.; and Li, B. 2025 · 2025

Closest in time.

Fiverr Freelance Services

Fiverr. 2025 · 2025

Closest in time.

Freelancer Marketplace

Freelancer. 2025 · 2025

Closest in time.

Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

Gemini Team, G. 2025 · 2025

Closest in time.

GitTaskBench: Anonymous GitHub Repository

GitTaskBench. 2025 · 2025