Hierarchical clustering based on single-pass for breaking topic detection and tracking

摘要

Single-pass is commonly used in topic detection and tracking ( TDT) due to its simplicity, high efficiency and low cost.When dealing with large-scale data, time cost will increase sharply and clustering performance will be affected greatly.Aiming at this problem, hierarchical clustering algo-rithm based on single-pass is proposed, which is inspired by hierarchical and concurrent ideas to di-vide clustering process into three stages.News reports are classified into different categories firstly. Then there are twice single-pass clustering processes in the same category, and one agglomerative clustering among different categories.In addition, for semantic similarity in news reports, topic model is improved based on named entities.Experimental results show that the proposed method can effectively accelerate the process as well as improve the performance.