Stream processing is a new memory computing paradigm that deals with dynamic data streams efficiently. Storm is one of the stream processing frameworks, but the default stream processing scheduler of storm also has some problems. For example, it does not consider reducing the cost in the cloud environment while ensuring the performance requirements. In this paper, a cost-efficient scheduling algorithm for Storm framework (CE-Storm) is proposed to reduce cost while satisfying deadline constrain. First, a new cost-efficient model (including resources usage cost, energy cost and communication cost) based on Storm framework is built. Then, based on the cost model, a cost-efficient scheduling algorithm which integrated resource monitoring module and communication detection module is designed. The nodes in the cluster are prioritized according to the cost-efficient information, and the nodes with higher priority are assigned tasks first to minimize the total cost of the cluster. Furthermore, this algorithm also reduces the communication cost between nodes and improves the cost effectiveness of the Storm cluster. We have performed extensive experiments on Storm clusters using Hibench's workloads in cloud environment. The result shows that the cost consumption of Storm clusters in cloud environment is reduced by 19.25 on average compared with the traditional scheduling algorithm. In others words, the proposed algorithms effectively improve the cost efficiency of Storm cluster in the cloud environment while satisfying the performance constrains.
展开▼
