BLADE: Benchmarking Language Model Agents for Data-Driven Science

Understand

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider.

• LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science.
• However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions.
• To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions.