Continuum Attention for Neural Operators

Understand

Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning.

• Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time series problems.
• Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators.
• Motivated by this, we study transformers in the function space setting.