Accelerating Neural Transformer via an Average Attention Network

Understand

With parallelizable attention networks, the neural Transformer is very fast to train.

• However, due to the auto-regressive architecture and self-attention in the decoder, the decoding procedure becomes slow.
• To alleviate this issue, we propose an average attention network as an alternative to the self-attention network in the decoder of the neural Transformer.
• The average attention network consists of two layers, with an average layer that models dependencies on previous positions and a gating layer that is stacked over the average layer to enhance the expressiveness of the proposed attention network.