No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval

Understand

Recent work has shown that small distilled language models are strong competitors to models that are orders of magnitude larger and slower in a wide range of information retrieval tasks.

• This has made distilled and dense models, due to latency constraints, the go-to choice for deployment in real-world retrieval applications.
• In this work, we question this practice by showing that the number of parameters and early query-document interaction play a significant role in the generalization ability of retrieval models.
• Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning.