Revisiting Linformer with a modified self-attention with linear complexity

Understand

Although Transformer models such as Google's BERT and OpenAI's GPT-3 are successful in many natural language processing tasks, training and deploying these models are costly and inefficient.Even if pre-trained models are used, deploying these models still remained a challenge due to their large size.

• Apart from deployment, these models take higher time during inference restricting user-friendliness.
• The main bottleneck is self-attention which uses quadratic time and space with respect to the sequence length.
• In order to reduce the quadratic time complexity of the self-attention mechanism, Linformer by Facebook's AI research team was introduced where they showed that the self-attention mechanism can be approximated by a low-rank matrix and exploiting this finding, a new method for self-attention with linear time and space complexity was proposed by them.