大数据环境下基于Hadoop框架的数据挖掘算法的研究与实现

摘要

In order to increase the speed of data mining for large data environment,analyze and study on distributed computing architecture Hadoop,put forward a kind of large data association rule mining algorithm based on Hadoop platform MRPrePost.In PrePost algorithm based on improved the algorithm,and reduce the difficulty of the distributed programming with Hadoop platform and easy to manage,through the depth of a bottom-up PrePost algorithm optimization strategy,reduce the memory overhead,at the same time using grouping strategy of load balancing,to improve the performance of parallel algorithm,the final test shows that the algorithm is fast,to adapt to the big data mining association rules.%为了提高大数据环境下的数据挖掘速度,对分布式计算构架Hadoop进行分析与研究,提出一种基于Hadoop平台的大数据关联规则挖掘算法MRPrePost.该算法在PrePost算法基础上改进而来,采用Hadoop平台降低分布式编程的难度且易于管理,通过一种自底向上的深度优化策略改进PrePost算法,降低内存开销,同时采用负载均衡的分组策略,来提高并行算法的性能,最终试验表明,该算法运行速度快,适应大数据关联规则挖掘.