Interpreting the Repeated Token Phenomenon in Large Language Models

Understand

Large Language Models (LLMs), despite their impressive capabilities, often fail to accurately repeat a single word when prompted to, and instead output unrelated text.

• This unexplained failure mode represents a vulnerability, allowing even end-users to diverge models away from their intended behavior.
• We aim to explain the causes for this phenomenon and link it to the concept of ``attention sinks'', an emergent LLM behavior crucial for fluency, in which the initial token receives disproportionately high attention scores.
• Our investigation identifies the neural circuit responsible for attention sinks and shows how long repetitions disrupt this circuit.