一种面向微博客文本流的噪音判别与内容相似性双重检测的过滤方法

摘要

微博客作为一种新的用户信息传播载体,在网络舆情发起和传播中起着重要作用.由于用户有意(上传广告)、无意(转发)操作所带来的大量噪音微博和相似微博,对网络舆情分析和用户浏览造成极为不利的影响.检测这些噪音微博和相似微博,对微博数据进行提纯,成为一个亟待解决的问题.基于统计数据分析了噪音微博和相似微博的特点,提出一种面向微博文本流的噪音判别和内容相似性双重检测的过滤方法:通过URL链接、字符率、高频词等特征判别,过滤噪音微博;通过分段过滤和索引过滤的双重内容过滤,检测和剔除相似微博.实验表明该方法能有效地对微博数据进行提纯,高效准确地过滤掉相似微博和噪音微博.%As a new carrier of user' s information dissemination, the microblog plays increasing important role in the emergence and propagation of Web public opinion. Large numbers of spam microblogs and similar microblogs caused by users' conscious ( uploading advertisements ) or unconscious (resending) operations have brought adverse effects on network public opinion analysis and users browsing. To test these spam microblogs and similar microblogs as well as to purify microblogs data become the problems to be urgently resolved. In this paper, the characteristics of spam microblogs and similar microblogs are analysed based on the statistical data, and a filtering approach for spam discrimination and content similarity double detection for microblog text stream is put forward. This method can filter spam microblogs through features discrimination in regard to URL links, character rate and high frequency words, and detect and eliminate similar microblogs through double content filtering: the subsection-based and the index-based. Experiments show that this method can effectively purify the microblogs and filter out similar microblogs and spam microblogs accurately.