Fetching the paper…
Reading the bibliography…
AI agents are an exciting new research direction, and agent development is driven by benchmarks.
On the Measure of Intelligence, November 2019
François Chollet · 1911
Earlier work this paper cites.
Artificial Intelligence: A Modern Approach
Stuart Jonathan Russell and Peter Norvig · 1995
Earlier work this paper cites.
Measuring Massive Multitask Language Understanding, January 2021
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt · 2009
Earlier work this paper cites.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, March 2021
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht · 2010
Earlier work this paper cites.
Machine Learning that Matters, June 2012
Kiri Wagstaff · 2012
Earlier work this paper cites.
50 Years of Data Science
David Donoho · 2017
Earlier work this paper cites.
Deep Reinforcement Learning That Matters
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger · 2018
Earlier work this paper cites.
Troubling Trends in Machine Learning Scholarship, July 2018
Zachary C. Lipton and Jacob Steinhardt · 2018
Earlier work this paper cites.
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, September 2018
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning · 2018
Earlier work this paper cites.
Optuna: A Next-generation Hyperparameter Optimization Framework
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama · 2019
Earlier work this paper cites.
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning, December 2019
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio · 2019
Earlier work this paper cites.
Reproducibility and Replicability in Science
National Academies of Sciences, Engineering, and Medicine · 2019
Earlier work this paper cites.
Mark Liberman - Reproducible Research and the Common Task Method (April 1, 2015), July 2019
Simons Foundation · 2019
Earlier work this paper cites.
Interactive Fiction Games: A Colossal Adventure
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan · 2020
Earlier work this paper cites.
Evaluating Large Language Models Trained on Code, July 2021
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021
Earlier work this paper cites.
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Omar Khattab, Christopher Potts, and Matei Zaharia · 2021
Earlier work this paper cites.
Competition-Level Code Generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022
Earlier work this paper cites.
ScienceWorld: Is your Agent Smarter than a 5th Grader?, November 2022
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu · 2022
Earlier work this paper cites.
Harms from Increasingly Agentic Algorithmic Systems
Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bhatt, Adrian Weller, David Krueger, and Tegan Maharaj · 2023
Earlier work this paper cites.
LLF-Bench: Benchmark for Interactive Learning from Language Feedback, December 2023
Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan · 2023
Earlier work this paper cites.
Large Language Models Cannot Self-Correct Reasoning Yet
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou · 2023
Earlier work this paper cites.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2023
Cited alongside, same era.
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, October 2023
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts · 2023
Cited alongside, same era.
Holistic Evaluation of Language Models, October 2023
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda · 2023
Cited alongside, same era.
The end of the “best open LLM”, April 2024
Nathan Lambert · 2024
Closest in time.
More Agents Is All You Need, February 2024
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye · 2024
Closest in time.
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents, January 2024
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He · 2024
Closest in time.
Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge · 2024
Closest in time.
Cheaper, Better, Faster, Stronger, April 2024
Mistral AI team · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang · 2023
Cited alongside, same era.
GAIA: a benchmark for General AI Assistants, November 2023
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom · 2023
Cited alongside, same era.
Practices for Governing Agentic AI Systems
Yonadav Shavit, Cullen O’Keefe, Tyna Eloundou, Paul McMillan, Sandhini Agarwal, Miles Brundage, Steven Adler, Rosie Campbell, Teddy Lee, Pamela Mishkin, Alan Hickey, Katarina Slama, Lama Ahmad, Alex Beutel, Alexandre Passos, and David G Robinson · 2023
Cited alongside, same era.
TaskBench: Benchmarking Large Language Models for Task Automation, December 2023
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang · 2023
Cited alongside, same era.
Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao · 2023
Cited alongside, same era.
Trading Off Compute in Training and Inference, July 2023
Pablo Villalobos and David Atkinson · 2023
Cited alongside, same era.
LLM Powered Autonomous Agents, June 2023
Lilian Weng · 2023
Cited alongside, same era.
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, February 2023
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan · 2023
Cited alongside, same era.
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, December 2023
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang · 2023
Cited alongside, same era.
Levels of AGI for Operationalizing Progress on the Path to AGI, June 2024
Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg · 2024
Closest in time.
Welcoming Diverse Approaches Keeps Machine Learning Strong, June 2024
Andrew Ng · 2024
Closest in time.
Identifying the Risks of LM Agents with an LM-Emulated Sandbox, May 2024
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto · 2024
Closest in time.
Can Language Models Solve Olympiad Programming?, April 2024
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao · 2024
Closest in time.
SteP: Stacked LLM Policies for Web Actions, April 2024
Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald · 2024
Closest in time.
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision, March 2024
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan · 2024
Closest in time.
OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, June 2024
OpenDevin Team · 2024
Closest in time.
On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models, May 2024
Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati · 2024
Closest in time.
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models, June 2024
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui · 2024
Closest in time.
TravelPlanner: A Benchmark for Real-World Planning with Language Agents, February 2024
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su · 2024
Closest in time.
SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press · 2024
Closest in time.
$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan · 2024
Closest in time.
The Shift from Models to Compound AI Systems, February 2024
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi · 2024
Closest in time.
LDB: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step, April 2024
Lily Zhong, Zilong Wang, and Jingbo Shang · 2024
Closest in time.
WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig · 2024
Closest in time.