Fetching the paper…
Reading the bibliography…
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering.
Discovering and exploring cases of educational source code plagiarism with Dolos, 2024
Rien Maertens, Maarten Van Neyghem, Maxiem Geldhof, Charlotte Van Petegem, Niko Strijbol, Peter Dawyndt, and Bart Mesuere · 2019
Earlier work this paper cites.
Openvaccine: Covid-19 mrna vaccine degradation prediction, 2020
Rhiju Das, H Wayment-Steele, Do Soon Kim, Christian Choe, Bojan Tunguz, Walter Reade, and Maggie Demkin · 2020
Earlier work this paper cites.
Program Synthesis with Large Language Models, August 2021
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton · 2021
Earlier work this paper cites.
Evaluating Large Language Models Trained on Code, July 2021
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021
Earlier work this paper cites.
Measuring Coding Challenge Competence With APPS, November 2021
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt · 2021
Earlier work this paper cites.
Research: quantifying GitHub Copilot’s impact on developer productivity and happiness, September 2022
Eirini Kalliamvakou · 2022
Earlier work this paper cites.
Competition-level code generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien De Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando De Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022
Earlier work this paper cites.
Anthropic’s Responsible Scaling Policy, Version 1.0, September 2023
Anthropic · 2023
Earlier work this paper cites.
Quantifying Memorization Across Neural Language Models, March 2023
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang · 2023
Earlier work this paper cites.
AgentBench: Evaluating LLMs as Agents, October 2023
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang · 2023
Earlier work this paper cites.
Vesuvius challenge - ink detection, 2023
Alex Lourenco, Brent Seales, Christy Chapman, Daniel Havir, Ian Janicki, JP Posma, Nat Friedman, Ryan Holbrook, Seth P., Stephen Parsons, and Will Cukierski · 2023
Cited alongside, same era.
GAIA: a benchmark for General AI Assistants, November 2023
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom · 2023
Cited alongside, same era.
Preparedness Framework, December 2023
OpenAI · 2023
Cited alongside, same era.
Can GPT-4 Perform Neural Architecture Search?, August 2023
Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie · 2023
Cited alongside, same era.
Cognition | Introducing Devin, the first AI software engineer, March 2024
cognition.ai · 2024
Cited alongside, same era.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica · 2024
Closest in time.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, April 2024
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2024
Closest in time.
DSBench: How Far Are Data Science Agents to Becoming Data Science Experts?, September 2024
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu · 2024
Closest in time.
Kaggle Progression System | Kaggle, 2024
Kaggle · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev · 2024
Cited alongside, same era.
GitHub Copilot Workspace: Welcome to the Copilot-native developer environment, April 2024
Thomas Dohmke · 2024
Cited alongside, same era.
Code Droid Technical Report, June 2024
factory.ai · 2024
Cited alongside, same era.
AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents, April 2024
Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, and Carolin Lawrence · 2024
Cited alongside, same era.
Frontier Safety Framework, May 2024
Google DeepMind · 2024
Cited alongside, same era.
AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation, May 2024a
Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui
Cited in the paper.
MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec
Cited in the paper.
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan · 2024
Closest in time.
Introducing Weco AIDE, April 2024
Dominik Schmidt, Zhengyao Jiang, and Yuxiang Wu · 2024
Closest in time.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein · 2024
Closest in time.
OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, 2024
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig · 2024
Closest in time.
The shift from models to compound ai systems, 2024
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi · 2024
Closest in time.
AutoCodeRover: Autonomous Program Improvement, July 2024
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury · 2024
Closest in time.