Efficient Attentions for Long Document Summarization

Understand

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization.

• In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source.
• We further conduct a systematic study of existing efficient self-attentions.
• Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions.