IDF Revisited: A Simple New Derivation within the Robertson-Sp?rck Jones Probabilistic Model

摘要

There have been a number of prior attempts to theoretically justify the effectiveness of the inverse document frequency (IDF). Those that take as their starting point Robertson and Sp?rck Jones's probabilistic model are based on strong or complex assumptions. We show that a more intuitively plausible assumption su ces. Moreover, the new assumption, while conceptually very simple, provides a solution to an estimation problem that had been deemed intractable by Robertson and Walker (1997).