Finding Inverse Document Frequency Information in BERT

Understand

For many decades, BM25 and its variants have been the dominant document retrieval approach, where their two underlying features are Term Frequency (TF) and Inverse Document Frequency (IDF).

• The traditional approach, however, is being rapidly replaced by Neural Ranking Models (NRMs) that can exploit semantic features.
• In this work, we consider BERT-based NRMs and study if IDF information is present in the NRMs.
• This simple question is interesting because IDF has been indispensable for the traditional lexical matching, but global features like IDF are not explicitly learned by neural language models including BERT.