Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling

Understand

Transformer is important for text modeling.

• However, it has difficulty in handling long documents due to the quadratic complexity with input text length.
• In order to handle this problem, we propose a hierarchical interactive Transformer (Hi-Transformer) for efficient and effective long document modeling.
• Hi-Transformer models documents in a hierarchical way, i.e., first learns sentence representations and then learns document representations.