Concept Drift Detection in Email Dataset through Intention Based Segmentation

摘要

In a dynamic environment, detection of changes in concept is very important for prediction and decision-based applications. Concept drift detection helps decision makers to perform smarter maintenance and operations at an appropriate time. In the context of Emails, concept drift is defined as how the concepts in Emails are changing with time. In this paper, a novel method is presented based on intention based segmentation to calculate concept drift in Email dataset. In intention based segmentation instead of comparing the contents of email as whole, it is divided into segments focusing on same intention and compared. Division of sentences is done on the basis of voices and tenses with the help of POS Tagging. After that, Hierarchical Clustering is used for clustering of segments and comparison is done using Vector Space Model. Importance of concept drift detection in Email domain is to find the emails which will be of less interest to user. It also helps to find the current interests of user by analyzing the changes in concepts over time.