Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency

Understand

Retrieval-augmented language models have demonstrated performance comparable to much larger models while requiring fewer computational resources.

• The effectiveness of these models crucially depends on the overlap between query and retrieved context, but the optimal degree of this overlap remains unexplored.
• In this paper, we systematically investigate how varying levels of query--context overlap affect model performance during both training and inference.
• Our experiments reveal that increased overlap initially has minimal effect, but substantially improves test-time perplexity and accelerates model learning above a critical threshold.