Spark并行化基于物品协同过滤算法

摘要

To solve the problem of time consumption and low efficiency in dealing with massive data using the traditional Item-Based algorithm,an Item-Based algorithm on Hadoop and Spark was proposed.Considering the execution efficiency and quality of the recommendation system,a music recommendation system was realized by improving the parallel Item-Based collaborative filtering algorithm on Spark.Results and quality of recommendation were provided for target user using the recommendation system through inputting some data from the KDD Cup music recommended competition.Experimental results show that the music recommendation system improves the execution efficiency and recommended quality.%针对传统的基于物品(Item-Based)协同过滤算法处理海量数据时耗时过长和效率低下问题,提出基于Hadoop分布式平台以及Spark并行计算模型的Item-Based协同过滤算法.综合考虑推荐系统的执行效率和推荐质量,通过对Item-Based协同过滤算法的改进,实现一个Spark并行化的音乐推荐系统.选取部分KDD Cup比赛数据集在推荐系统中进行测试,为目标用户生成音乐推荐结果和评定推荐误差,实验结果表明,改进后的算法在执行效率和推荐质量方面有了显著提高.