WebSuite: Systematically Evaluating Why Web Agents Fail

Understand

We describe WebSuite, the first diagnostic benchmark for generalist web agents, designed to systematically evaluate why agents fail.

• Advances in AI have led to the rise of numerous web agents that autonomously operate a browser to complete tasks.
• However, most existing benchmarks focus on strictly measuring whether an agent can or cannot complete a task, without giving insight on why.
• In this paper, we 1) develop a taxonomy of web actions to facilitate identifying common failure patterns, and 2) create an extensible benchmark suite to assess agents' performance on our taxonomized actions.