Analysis of emails using a hidden Markov model to recognize sections of the email, e.g. header, body, signature block and disclaimer

摘要

An automated parser for e-mail messages identifies component parts such as header, body, signature, and disclaimer. The parser uses a hidden Markov model (HMM) in which the lines making up an e mail are treated as a sequence of observations of a system that evolves according to a Markov chain having states corresponding to the component parts. The HMM is trained using a manually-annotated set of e-mail messages, then applied to parse other e-mail messages. HMM-based parsing can be further refined or expanded using heuristic post-processing techniques that exploit redundancy of some component parts (e.g., signatures, disclaimers) across a corpus of e-mail messages, e.g. to cluster email according to the similarity of signature block and to compare the similarly within clusters to find a representative signature for each cluster.