Using Frequent Max Substring Technique for Thai Text Indexing

摘要

This paper proposes a Thai texts indexing method using a frequent max substring technique to improve the efficiency of indexing Thai texts. Thai texts are considered as un-delimited language where the structure of writing is a string of symbols without explicit word delimiters. Therefore, some pre-processing technique may need to be applied to discover important patterns before indexing can be performed. In this paper, the frequent max substring technique is proposed as a promising alternative for indexing Thai texts. The proposed technique extracts indexing terms as long and frequent substrings, called frequent max substrings, from Thai texts. This method is used to extract the patterns of interest without context consideration and is interested in substrings that occur frequently in Thai texts in order to reduce the number of insignificant indexing terms. It is also language-independent and does not rely on any dictionary or language grammatical knowledge. The new data structure, called Frequent Suffix Trie, is also proposed to assure exhaustive enumeration of substrings to support extracting the frequent max substrings. The frequent max substrings are then used as indexing terms, together with their number of occurrences and positions, to form an index. To illustrate the proposed technique, experimental studies and comparison results on indexing Thai texts are presented in this paper. The results show that the frequent max substring technique provide a more efficient way for storing indexing terms: by indexing only the frequent max substrings.