HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization

Understand

To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model.

• However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents.
• To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization.
• Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer.