State Spaces Aren't Enough: Machine Translation Needs Attention

Understand

Structured State Spaces for Sequences (S4) is a recently proposed sequence model with successful applications in various tasks, e.g.

• vision, language modeling, and audio.
• Thanks to its mathematical formulation, it compresses its input to a single hidden state, and is able to capture long range dependencies while avoiding the need for an attention mechanism.
• In this work, we apply S4 to Machine Translation (MT), and evaluate several encoder-decoder variants on WMT'14 and WMT'16.