Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow

Understand

Language models are known to absorb biases from their training data, leading to predictions driven by statistical regularities rather than semantic relevance.

• We investigate the impact of these biases on answer choice preferences in the Massive Multi-Task Language Understanding (MMLU) task.
• Our findings show that these biases are predictive of model preference and mirror human test-taking strategies even when chain of thought (CoT) reasoning is used.
• To address this issue, we introduce Counterfactual Prompting with Agnostically Primed CoT (APriCoT).