An Efficient Transformer Decoder with Compressed Sub-layers

Understand

The large attention-based encoder-decoder network (Transformer) has become prevailing recently due to its effectiveness.

• But the high computation complexity of its decoder raises the inefficiency issue.
• By examining the mathematic formulation of the decoder, we show that under some mild conditions, the architecture could be simplified by compressing its sub-layers, the basic building block of Transformer, and achieves a higher parallelism.
• We thereby propose Compressed Attention Network, whose decoder layer consists of only one sub-layer instead of three.