Music Listening History Dataset Curation and Distributed Music Recommendation Engines Using Collaborative Filtering

摘要

In this paper, a new system for creation of music recommendation engines based on massively large datasets containing music listening patterns for users is introduced. The engine is designed to be scalable and hence, uses distributed computing clusters to perform tasks and calculations. The collaborative filtering algorithm is implemented on a distributed Apache Spark cluster to maximize performance and efficiency. Also, the open source Creative Commons (CC0) licensed ListenBrainz dataset is introduced. It makes public for commercial and non-commercial usage the music listening histories of over 4500 users and counting. The dataset includes over 180 million unique listening records (called listens) from various sources with additional metadata including MusicBrainz IDs which link the listens to additional sources of information. The collection of such music listening data is very useful for industry and research as open data in large quantities is not easily available. A system is developed to allow continuous access to this data and to release snapshots of the data regularly. The aforementioned music recommendation engine then uses this data to train its models and create recommendations for users.