Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines

Understand

Despite their ubiquity in core AI fields like natural language processing, the mechanics of deep attention-based neural networks like the Transformer model are not fully understood.

• In this article, we present a new perspective towards understanding how Transformers work.
• In particular, we show that the "dot-product attention" that is the core of the Transformer's operation can be characterized as a kernel learning method on a pair of Banach spaces.
• In particular, the Transformer's kernel is characterized as having an infinite feature dimension.