Shallow pooling for sparse labels

Understand

Recent years have seen enormous gains in core IR tasks, including document and passage ranking.

• Datasets and leaderboards, and in particular the MS MARCO datasets, illustrate the dramatic improvements achieved by modern neural rankers.
• When compared with traditional test collections, the MS MARCO datasets employ substantially more queries with substantially fewer known relevant items per query.
• Given the sparsity of these relevance labels, the MS MARCO leaderboards track improvements with mean reciprocal rank (MRR).