BERTology Meets Biology: Interpreting Attention in Protein Language Models

Understand

Transformer architectures have proven to learn useful representations for protein classification and generation tasks.

• However, these representations present challenges in interpretability.
• In this work, we demonstrate a set of methods for analyzing protein Transformer models through the lens of attention.
• We show that attention: (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth.