Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains

Understand

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages.

• To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to study them.
• A key finding is that when trained on first-order Markov chains, transformers with two or more layers consistently develop an induction head mechanism to estimate the in-context bigram conditional distribution.
• In contrast, single-layer transformers, unable to form an induction head, directly learn the Markov kernel but often face a surprising challenge: they become trapped in local minima representing the unigram distribution, whereas deeper models reliably converge to the ground-truth bigram.