LongFuncEval: Measuring the effectiveness of long context models for function calling

Understand

Multiple recent studies have documented large language models' (LLMs) performance on calling external tools/functions.

• Others focused on LLMs' abilities to handle longer context lengths.
• At the intersection of these areas lies another interesting problem: LLMs' abilities to accurately perform function calls in long context settings.
• Particularly, when calling tools, LLMs are encumbered by three predominant challenges: (1) a large catalog of tools, (2) long responses from the tool APIs, and (3) long multi-turn conversations.