The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models

Understand

Following multiple instructions is a crucial ability for large language models (LLMs).

• Evaluating this ability comes with significant challenges: (i) limited coherence between multiple instructions, (ii) positional bias where the order of instructions affects model performance, and (iii) a lack of objectively verifiable tasks.
• To address these issues, we introduce a benchmark designed to evaluate models' abilities to follow multiple instructions through sequential instruction following (SIFo) tasks.
• In SIFo, the successful completion of multiple instructions is verifiable by examining only the final instruction.