DupM: a Data Replica Allocation Strategy for Distributed Mining

摘要

The mass data mining algorithm is usually sensitive to computing resources, the quantities of which (e.g., the quantities of memory and CPUs available for mining algorithms) determine the mining results. In this paper, data replicas as a resource were incorporated into the resource scheduling strategy for distributed data mining to offer a replica-aware resource scheduling strategy DupM. With data replicas as a type of data resources, the DupM strategy scheduled data replicas through dynamic programming so that a cloud platform could, based on the requirements of distributed mining, allocate replicas. The simulation test of test data sets in KDD CUP events and IBM synthesizer data sets demonstrated that the DupM resource scheduling strategy has more advantages than Hadoop’s built-in resource scheduling strategy.