ACEBench: Who Wins the Match Point in Tool Usage?

Understand

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems.

• However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead.
• To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs.
• ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent.