Adaptive Term Frequency Normalization for BM25

摘要

A key component of BM25 contributing to its success is its sub-linear term frequency (TF) normalization formula. The scale and shape of this TF normalization component is controlled by a parameter k_1, which is generally set to a term-independent constant. We hypothesize and show empirically that in order to optimize retrieval performance, this parameter should be set in a term-specific way. Following this intuition, we propose an information gain measure to directly estimate the contributions of repeated term occurrences, which is then exploited to fit the BM25 function to predict a term-specific k_1. Our experiment results show that the proposed approach, without needing any training data. can efficiently and automatically estimate a term-specific k_1, and is more effective and robust than the standard BM25.