Improvement of recommendation algorithm based on Collaborative Deep Learning and its Parallelization on Spark

摘要

Collaborative Deep Learning (CDL) utilizes the strong feature learning capability of neural network and the model fitting robustness to solve the problem that the performance of Recommender System drops dramatically when the data is sparse. However, it makes the model training become difficult to maintain when Recommender System faces a large amount of data, and a variety of unpredictable problems will arise. In order to solve the above problems, collaborative deep learning and its parallelization methods were studied in this study, and an improved model CDL-I (CDL with item private node) aiming at item content optimization based on collaborative deep learning was proposed, which improved SDAE on the basis of CDL, added private network nodes; in case of sharing the network parameters of the model, private bias terms were added for each item. As a result, the network may learn the item content parameters in a more targeted manner, thereby enhancing the detection performance of the model on item content in Recommender System. Furthermore, the algorithm was parallelized by splitting the model, and a parallel training CDL-I method was also proposed, which was transplanted to the Spark distributed cluster. The parameters of each part of the model were trained and optimized in parallel to enhance the scale and scalability of data that the model could process. The experiments on multiple real datasets have verified the effectiveness and efficiency of the proposed parallel CDL-I algorithm.