Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding

Understand

The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity.

• Many works focus on approximating the dot-then-exponentiate softmax function in the original attention, leading to sub-quadratic or even linear-complexity Transformer architectures.
• However, we show that these methods cannot be applied to more powerful attention modules that go beyond the dot-then-exponentiate style, e.g., Transformers with relative positional encoding (RPE).
• Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.