Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations

Understand

Chain-of-thought explanations are widely used to inspect the decision process of large language models (LLMs) and to evaluate the trustworthiness of model outputs, making them important for effective collaboration between LLMs and humans.

• We demonstrate that preference optimization - a key step in the alignment phase - can inadvertently reduce the faithfulness of these explanations.
• This occurs because the reward model (RM), which guides alignment, is tasked with optimizing both the expected quality of the response and the appropriateness of the explanations (e.g., minimizing bias or adhering to safety standards), creating potential conflicts.
• The RM lacks a mechanism to assess the consistency between the model's internal decision process and the generated explanation.