True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4

Understand

Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks.

• This calls for a more challenging benchmark requiring highly advanced reasoning ability to be solved.
• In this paper, we introduce such a benchmark, consisting of 191 long-form (1200 words on average) mystery narratives constructed as detective puzzles.
• Puzzles are sourced from the "5 Minute Mystery" platform and include a multiple-choice question for evaluation.