ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows

Imagenette

Jeremy Howard and Others · 2019

Earlier work this paper cites.

Program synthesis with large language models, 2021

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton · 2021

Earlier work this paper cites.

React: Synergizing reasoning and acting in language models, 2023

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao · 2023

Earlier work this paper cites.

Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry · 2024

Cited alongside, same era.

Mlagentbench: Evaluating language agents on machine learning experimentation, 2024

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec · 2024

Cited alongside, same era.

Swe-bench: Can language models resolve real-world github issues?, 2024

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2024

Cited alongside, same era.

Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024

Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein · 2024

Later among the works it cites.

Openhands: An open platform for ai software developers as generalist agents, 2024

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig · 2024

Later among the works it cites.