Automated Text Classification System Based on Statistical Unified Model

摘要

The paper is devoted to the automated text classification system based on a unified model. The main text mining statistical and linguistic approaches are considered. The architecture of the developed system is given. The automated system is analyzed in details. The text data unified model consists of statistical elementary models: substrings, cumulative and the finite difference characteristics. The structure and features of each model are considered. The unified model flexibility is achieved by assigning weights for each elementary model. According to this logic, the system can be modified for different text types. The automated system for literary text classification has been tested. The classification quality was evaluated by precision, recall and f-measure. The model was trained and evaluated on the 6 classes. The total texts number in training set and test set is 600 and 60 respectively. The automated text classification system shows good results, low scores for some texts are explained. The advantages and limitations of the proposed system are shown. In addition, there is research area on the linguistic models inclusion in order to improve the classification quality of the proposed automated system.