Long Range Arena: A Benchmark for Efficient Transformers

Understand

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity.

• In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
• To this date, there is no well-established consensus on how to evaluate this class of models.
• Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models.