Understand
Powered by large language models (LLMs), web browsing agents operate graphical user interfaces in a human-like manner, offering a transparent and general framework for automating web-based tasks.
- As these agents rapidly improve and achieve strong performance on existing benchmarks such as WebArena, a key question arises: $\textit{Can current benchmarks still accurately evaluate the capabilities of increasingly powerful agents, especially for more tedious and cognitively demanding tasks?}$ In this paper, we present $\textbf{WebChoreArena}$, a substantial extension of WebArena designed to push beyond general browsing scenarios.
- WebChoreArena introduces 532 carefully curated tasks developed over 300+ hours, explicitly targeting more labor-intensive and complex web chores.
- It systematically expands the evaluation space along three critical dimensions: (i) $\textbf{Massive Memory}$, requiring agents to accurately retain and retrieve large amounts of information from observations; (ii) $\textbf{Calculation}$, demanding precise mathematical reasoning over collected information; and (iii) $\textbf{Long-Term Memory}$, necessitating consistent information tracking across multiple webpages.