Massive Online Shopping Data Mining based on Hadoop

摘要

Huge volume data generated by human interacting with internet contains useful hidden information, e.g. the online shopping data in electronic commerce platform that reflect people's interests and requirements. To exploit the potential value of massive online shopping data and improve the service of online retailers, this paper proposes an optimized distributed Apriori algorithm using Hadoop platform. The data for the experiment are from the Tianchi data laboratory. In the experiments, we examined the performance of the proposed approach against different number of nodes in Hadoop cluster. Besides, the goods with the highest correlation degree among the massive online shopping data are investigated. The results showed that the optimized Apriori algorithm has the capability to tackle the online shopping data mining task. In particular, increasing the number of nodes of the cluster appropriately can decrease the running time. In addition, when the 120 million online shopping records are used for the experiment, we found that the two goods numbered as 50010200 and 50008168 have the highest correlation degree and the probability reaches 0.00758%. In a total of more than 120 million massive records, this is a relatively high frequency.