Modelling and programming stream-based distributed computing based on the meta-pipeline approach

摘要

Caravela is a computing platform that has been designed in academia to implement distributed stream-based computing based on the flow-model. This paper is focused on the new meta-pipeline execution mechanism, which allows to set up a pipeline of flow-models to be executed in parallel but with data streams flowing in sequence along them. A model and programming tools are presented to develop the proposed meta-pipeline in the Caravela platform. These tools are important, namely to tackle the deadlock problem that can occur due to uninitialised input data streams. A new efficient algorithm is also proposed to identify off-line the uninitialised edges in a 'pipeline graph'. Moreover, optimisation techniques are also proposed to reduce the computation time and the amount of memory required to implement the proposed algorithm, and a graphical user interface (GUI) was implemented to facilitate the programming of applications with the meta-pipeline approach. The experimental results presented in this paper show the practical interest of the meta-pipeline approach for stream-based computing and the effectiveness of both the proposed algorithm and the developed GUI for programming the Caravela platform.