ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions

Understand

Large language models (LLMs) have demonstrated strong capabilities in using external tools to address user inquiries.

• However, most existing evaluations assume tool use in short contexts, offering limited insight into model behavior during realistic long-term interactions.
• To fill this gap, we introduce ToolHaystack, a benchmark for testing the tool use capabilities in long-term interactions.
• Each test instance in ToolHaystack includes multiple tasks execution contexts and realistic noise within a continuous conversation, enabling assessment of how well models maintain context and handle various disruptions.