Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity

Understand

Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools.

• However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both.
• In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles.
• To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference.