Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps

Understand

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction.

• Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs.
• We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework.
• FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction.