Clustering Short Text using a Centroid-Based Lexical Clustering Algorithm

摘要

Traditional lexical clustering methods process text as a bag of words, with similarity between two test-fragments measured on the basis of word co-occurrence. While this approach is suitable for clustering large fragments of text (e.g., documents), it performs poorly when clustering smaller text fragments such as sentences (e.g., short text or quotes). This is because two sentences may be semantically similar while containing no common words. This paper proposes a new variant of the standard k-means algorithm for short text clustering that is based on the notion of synonym expansion semantic vectors. These vectors represent short text using semantic information derived from a lexical database constructed to identify the correct meaning to a word, based on the context in which it appears. Thus, whereas conventional it-means algorithm application is based on measuring the distance between patterns, the proposed approach is based on measuring semantic similarity between patterns (e, g., sentences). This enables it to utilise a higher degree of semantic information available within the clustered sentences. Empirical results show that the proposed variant method performs favorably against other clustering technique on two specially constructed datasets of famous quotations, benchmark datasets in several other domains, and that its incorporation as a short text similarity using synonym expansion leads to a significant improvement in the centroid-based clustering performance. Therefore, it is potential use in a variety of knowledge discovery processing tasks including text summarisation and text mining.