Hierarchical Transformers Are More Efficient Language Models

Understand

Transformer models yield impressive results on many NLP and sequence modeling tasks.

• Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or well-structured images produced by DALL-E.
• These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility.
• We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences.