Analyzing Transformers in Embedding Space

Understand

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning.

• While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks.
• In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on.
• We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity.