Overthinking the Truth: Understanding how Language Models Process False Demonstrations

Understand

Modern language models can imitate complex patterns through few-shot learning, enabling them to complete challenging tasks without fine-tuning.

• However, imitation can also lead models to reproduce inaccuracies or harmful content if present in the context.
• We study harmful imitation through the lens of a model's internal representations, and identify two related phenomena: "overthinking" and "false induction heads".
• The first phenomenon, overthinking, appears when we decode predictions from intermediate layers, given correct vs.