Hard-Coded Gaussian Attention for Neural Machine Translation

Understand

Recent work has questioned the importance of the Transformer's multi-headed attention for achieving high translation quality.

• We push further in this direction by developing a "hard-coded" attention variant without any learned parameters.
• Surprisingly, replacing all learned self-attention heads in the encoder and decoder with fixed, input-agnostic Gaussian distributions minimally impacts BLEU scores across four different language pairs.
• However, additionally hard-coding cross attention (which connects the decoder to the encoder) significantly lowers BLEU, suggesting that it is more important than self-attention.