自适应信息过滤中使用少量正例进行阈值优化

摘要

One special challenge in adaptive information filtering is the problem of extremely sparse data. So it is very important to learn optimal threshold while filtering the input textual stream. In this paper, an algorithm is presented for the threshold optimization. The algorithm learns fast by using few positive samples. Moreover, most of the quantities the algorithm requires can be updated incrementally, so its memory and computational power requirements are low. It also has the merits of effective, robust, and practically useful. Fudan University's adaptive text filtering system used this algorithm for the first time and came in third in all runs of TREC10. Its T10U and T10F are 0.215 and 0.414 respectively.%自适应信息过滤中一个大的挑战在于其数据稀疏问题.因此,在对输入的文本流进行过滤的同时学习最优阈值非常重要.提出了一种新颖的阈值优化算法.该算法可以通过少量的正例进行快速的学习,所需数据的获得具有增量性,故而其计算量及所需的存储空间很小.此外,该算法还具有高效、健壮、实用性强等优点.在第10届国际文本检索会议(TREC10)上,复旦大学的自适应信息过滤系统使用了该阈值优化算法,并取得了第3名的成绩.其T10U和T10F分别达到了0.215和0.414.