Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

Understand

Transformer-based models have brought a radical change to neural machine translation.

• A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input.
• However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns.
• In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed -- non-learnable -- attentive patterns that are solely based on position and do not require any external knowledge.