首页>
外国专利>
Automated load-balancing of partitions in arbitrarily imbalanced distributed mapreduce computations
Automated load-balancing of partitions in arbitrarily imbalanced distributed mapreduce computations
展开▼
机译:任意不平衡分布式mapreduce计算中分区的自动负载均衡
展开▼
页面导航
摘要
著录项
相似文献
摘要
A distributed computing system executes a MapReduce job on streamed data that includes an arbitrary amount of imbalance with respect to the frequency distribution of the data keys in the dataset. A map task module maps the dataset to a coarse partitioning, and generates a list of the top K keys with the highest frequency among the dataset. A sort task module employs a plurality of sorters to read the coarse partitioning and sort the data into buckets by data key. The values for the top K most frequent keys are separated into single-key buckets. The other less frequently occurring keys are assigned to buckets that each have multiple keys assigned to it. Then, more than one worker is assigned to each single-key bucket. The output of the multiple workers assigned to each respective single-key bucket is stitched together.
展开▼