LIDER: An Efficient High-dimensional Learned Index for Large-scale Dense Passage Retrieval

Understand

Many recent approaches of passage retrieval are using dense embeddings generated from deep neural models, called "dense passage retrieval".

• The state-of-the-art end-to-end dense passage retrieval systems normally deploy a deep neural model followed by an approximate nearest neighbor (ANN) search module.
• The model generates embeddings of the corpus and queries, which are then indexed and searched by the high-performance ANN module.
• With the increasing data scale, the ANN module unavoidably becomes the bottleneck on efficiency.