Coding Agents with Multimodal Browsing are Generalist Problem Solvers

Captcha: Using hard ai problems for security

Luis von Ahn, Manuel Blum, Nicholas J. Hopper, and John Langford · 2003

Earlier work this paper cites.

Gaia: a benchmark for general ai assistants

Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom · 2023

Earlier work this paper cites.

Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023

Earlier work this paper cites.

Webarena: A realistic web environment for building autonomous agents

Original

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al · 2023

Earlier work this paper cites.

Tapeagents: a holistic framework for agent development and optimization

Original

Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, et al · 2024

Earlier work this paper cites.

Magentic-one: A generalist multi-agent system for solving complex tasks

Original

Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al · 2024

Earlier work this paper cites.

SWE-bench: Can language models resolve real-world github issues?

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan · 2024

Earlier work this paper cites.

Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

Original

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried · 2024

Earlier work this paper cites.

Browser use: Enable ai to control your browser, 2024

Magnus Müller and Gregor Žunič · 2024

Cited alongside, same era.

Openhands: An open platform for ai software developers as generalist agents

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al · 2024

Cited alongside, same era.

Agent workflow memory

Original

Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig · 2024

Cited alongside, same era.

Agentless: Demystifying llm-based software engineering agents

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang · 2024

Cited alongside, same era.

Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu · 2024

Cited alongside, same era.

The "think" tool: Enabling claude to stop and think in complex tool use situations

Anthropic(2025) · 2025

Closest in time.

The browsergym ecosystem for web agent research

Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados · 2025

Closest in time.

Agentless-lite

Dunn(2025) · 2025

Closest in time.

Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025

Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li · 2025

Closest in time.

Open-source deepresearch – freeing our search agents, 2025

Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier · 2025

Closest in time.