Scalable Transformers for Neural Machine Translation

Understand

Transformer has been widely adopted in Neural Machine Translation (NMT) because of its large capacity and parallel training of sequence generation.

• However, the deployment of Transformer is challenging because different scenarios require models of different complexities and scales.
• Naively training multiple Transformers is redundant in terms of both computation and memory.
• In this paper, we propose a novel Scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters.