Survey on Evaluation of LLM-based Agents

Understand

LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments.

• This paper provides the first comprehensive survey of evaluation methods for these increasingly capable agents.
• We analyze the field of agent evaluation across five perspectives: (1) Core LLM capabilities needed for agentic workflows, like planning, and tool use; (2) Application-specific benchmarks such as web and SWE agents; (3) Evaluation of generalist agents; (4) Analysis of agent benchmarks' core dimensions; and (5) Evaluation frameworks and tools for agent developers.
• Our analysis reveals current trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks.