IncDSI: Incrementally Updatable Document Retrieval

Understand

Differentiable Search Index is a recently proposed paradigm for document retrieval, that encodes information about a corpus of documents within the parameters of a neural network and directly maps queries to corresponding documents.

• These models have achieved state-of-the-art performances for document retrieval across many benchmarks.
• These kinds of models have a significant limitation: it is not easy to add new documents after a model is trained.
• We propose IncDSI, a method to add documents in real time (about 20-50ms per document), without retraining the model on the entire dataset (or even parts thereof).