From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers

Understand

Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation.

• In this work, we study learning a 1-layer self-attention model from a set of prompts and associated output data sampled from the model.
• We first establish a precise mapping between the self-attention mechanism and Markov models: Inputting a prompt to the model samples the output token according to a context-conditioned Markov chain (CCMC) which weights the transition matrix of a base Markov chain.
• Additionally, incorporating positional encoding results in position-dependent scaling of the transition probabilities.