Task Allocation in Distributed Mesh-Connected Machine Learning System: Simplified Busy List Algorithm with Q-Learning Based Queuing

摘要

In the era where organizations gather and process more and more data, Machine Learning (ML) techniques become increasingly important. Considering “Big Data,” ML usually involves intensive data processing and high-performance computing. To meet the growing requirements, efficient distributed and parallel systems are key factors. In this paper, we consider mesh-based distributed system and task allocation methods in the system. We focus especially on the impact of intelligent queuing in task allocation algorithms. A new SBL algorithm with Q-Learning queuing is presented. In addition, the new SBL technique is compared to other wellknown allocation schemes, which are discussed as well. The comparison is made using an implemented experimentation system and simulation results are presented. The results confirm that SBL algorithm and the queuing system deliver good performance characteristic.