Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers

Understand

Vision transformers have delivered tremendous success in representation learning.

• This is primarily due to effective token mixing through self attention.
• However, this scales quadratically with the number of pixels, which becomes infeasible for high-resolution inputs.
• To cope with this challenge, we propose Adaptive Fourier Neural Operator (AFNO) as an efficient token mixer that learns to mix in the Fourier domain.