Sharing Attention Weights for Fast Transformer

Understand

Recently, the Transformer machine translation system has shown strong results by stacking attention layers on both the source and target-language sides.

• But the inference of this model is slow due to the heavy use of dot-product attention in auto-regressive decoding.
• In this paper we speed up Transformer via a fast and lightweight attention model.
• More specifically, we share attention weights in adjacent layers and enable the efficient re-use of hidden states in a vertical manner.