Local Self-Attention over Long Text for Efficient Document Retrieval

Understand

Neural networks, particularly Transformer-based architectures, have achieved significant performance improvements on several retrieval benchmarks.

• When the items being retrieved are documents, the time and memory cost of employing Transformers over a full sequence of document terms can be prohibitive.
• A popular strategy involves considering only the first n terms of the document.
• This can, however, result in a biased system that under retrieves longer documents.