A MapReduce Reinforced Distributed Sequential Pattern Mining Algorithm

摘要

Redesign and reimplementation of traditional sequential pattern mining algorithms on distributed computing frameworks are essential for dealing with big data. Along the way, the critical issue is how to minimize the communication overhead of the distributed sequential pattern mining algorithm and maximize its execution efficiency by balancing the workload of distributed computing resources. To address such an issue, this paper proposes a MapReduce reinforced distributed sequential pattern mining algorithm DGSP (Distributed GSP algorithm based on MapReduce), which consists of two MapReduce jobs. The "two-jobs" structure of DGSP can effectively reduce the communication overhead of the distributed sequential pattern mining algorithm. DGSP also enables optimizing the workload balance and the execution efficiency of distributed sequential pattern mining by evenly partitioning the database and assigning the fragments to Map workers. Experimental results indicate that DGSP can significantly improve the overall performance, scalability and fault tolerance of sequential pattern mining on big data.