Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

Understand

Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years.

• However, most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system.
• In this work, we introduce a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers.
• Fast-forward connections play an essential role in propagating the gradients and building a deep topology of depth 16.