Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection

Understand

Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks.

• Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context.
• To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}.
• Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments.