Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

Understand

Transformer models have achieved state-of-the-art results across a diverse range of domains.

• However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow.
• In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed.
• However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models.