Keyword sequence extraction based on byte entropy iterative segmentation

摘要

For the problem of mixed unknown message classification, variable field in message causes difficulty in keyword sequence extraction and message analysis. In this paper, we propose a message classification and keyword sequence extraction method based on byte entropy iterative segmentation (BEIS). In BEIS, firstly, we divide the messages into coarse clusters and extract the first keyword from each cluster. Secondly, we find the rest keywords in an iterative way. For each turn of a keyword extraction, we define links (two adjacent bytes) to find the keyword's offset in messages according to the frequency of links, and estimate the length of the keyword via byte entropy trend, the rest part excluding the extracted keyword being the object for next iterative extraction. Experimental results show that compared with relative existing algorithms, BEIS is more accurate and stable.