When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

Understand

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists.

• To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts.
• In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}.
• To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators.