TASK ALLOCATION METHOD AND TASK ALLOCATION APPARATUS FOR DISTRIBUTED DATA CALCULATION

摘要

The present application discloses a task allocation method and task allocation apparatus for distributed data computing. The task allocation method includes: receiving storage parameters for target data to be computed in distributed data; mapping data slices of the target data to a resilient distributed dataset based on the storage parameters, each data slice corresponding respectively to a partition in the resilient distributed dataset; assigning each partition to a storage node to generate a computing task and perform the computing tasks. By using data storage information in a distributed database to allocate computing tasks to storage nodes corresponding to the data, Only data in local memories need to be called during the computing process, IO redundancy and time consumed due to repeated data forwarding are reduced.