Traditional mining based on parallel Apriori algorithms needs much more time in data IO with the increasing size of large transaction database.In this paper,we improved Apriori algorithm in three aspects: compression in the transaction,reducing the number of scanning areas,and simplifying the candidate set generation.We proposed "0" and "1" as the entries to describe the transaction Boolean matrix model,and introduced the weight dimensions to compress the matrix size of the transaction.Meanwhile,dynamic pruning matrix is adopted,and "and" operation of matrix is applied to generate a candidate set.The experiments of the improved algorithm running parallel in Hadoop framework show that the algorithm is suitable for large-scale data mining,and the algorithm has good scalability and effectiveness.%对于规模庞大的事务数据库,传统的并行Apriori算法在挖掘中会在数据IO上有较大的时间开销.从压缩事务、减少扫描次数、简化候选集生成3 个方面对Apriori 算法进行改进.提出了以元素"0"和"1"表示事务的布尔矩阵模型,并引入权值维度,压缩了相同事务的矩阵规模.同时,动态地进行剪枝,矩阵的"与"运算用于候选集合的生成.将改进后的算法在Hadoop 框架上进行并行化实现,实验表明该算法适合大规模数据挖掘且具有良好的伸缩性与有效性.
展开▼
