Impact of I/O and execution scheduling strategies on large scale parallel data mining

摘要

In the era of “Big Data”, there is an emerging need to process a massive data set using large cluster system. Anyway, without the right strategies to handle the data, it is challenging to gain a good performance from the system. In this paper, many I/O and execution scheduling strategies for parallel data mining application has been investigated. The goal is to discover strategies that balance the data processing load and better utilize a multi-core cluster system for data mining application. Issues that impact the performance have been explored. The simulation results show that a substantial performance improvement can be obtained especially with a multi-core cluster system when a proper I/O and task execution sequence scheduling has been employed.