CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Understand

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models.

• We present the data set design and benchmark that supports scoring against a crowd-validated human baseline.
• We test six state-of-the-art models against our benchmark.
• Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.