General-purpose, long-context autoregressive modeling with Perceiver AR

Understand

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression.

• However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure.
• We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking.
• Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms.