EFFICIENT IMPLEMENTATION OF LARGE-SCALE WORKFLOWS BASED ON ARRAY CONTRACTION

摘要

The size of a workflow representation, used in programming languages and runtime systems, depends on the number of included tasks and their connections. Therefore, the execution of large-scale workflows is limited by memory size of the master node and task scheduling/transfer cost. We propose a scheme largely reducing the size of workflow representation using array contraction. Focusing on arrays in workflow representation, our scheme can contract such arrays dynamically, without static analysis of user code. Hierarchically parallel structures, often used in large-scale workflows, can also be contracted. As a result of evaluation on our object-oriented workflow language MegaScript, the number of API objects in fully-contracted random workflow representations was approximately 300 in average, independent from the number of tasks. The required memory size was also reduced to approximately 100KB in average.