Performance Evaluation of Apriori Algorithm on a Hadoop Cluster

摘要

Frequent Itemset Mining is a well-known concept in data sciences. If we feed frequent itemset miner algorithms with large datasets they become resource hungry fast as their search space explodes. This problem is even more apparent when we try to use them on Big Data. Recent advances in parallel programming provides good solutions to deal with large datasets but they present their own problems when we try to modify existing data mining algorithms for the new paradigms. The Apriori-algorithm is a classic solution for mining frequent item-sets. In this paper, we provide a parallel implementation of the Apriori algorithm for the Hadoop platform. We introduce a method to measure the performance of the distributed algorithm. In our experimental results we find choke points in the algorithm and provide resolutions.