Quantifying Attention Flow in Transformers

Understand

In the Transformer model, "self-attention" combines information from attended embeddings into the representation of the focal embedding in the next layer.

• Thus, across layers of the Transformer, information originating from different tokens gets increasingly mixed.
• This makes attention weights unreliable as explanations probes.
• In this paper, we consider the problem of quantifying this flow of information through self-attention.