Towards a Workflow-Aware Distributed Versioning File System for Metacomputing Systems

摘要

Traditional distributed file systems (DFSs) are often adopted as backplanes to facilitate the data accesses of workflow-based computations in metacomputing systems. For example, during the computation of each job, files may be read from previously completed jobs and written to be consumed by the later ones according to data dependencies. However, traditional DFSs, which are built on top of local tree-structured file systems, are not targeted at this particular computing environment. For example, in practice, the data dependency, also named dataflow information, is quite useful for the scheduler to effectively exploit the concurrency among the jobs during the scheduling process. Unfortunately, traditional DFSs have no effective ways to track such information, leaving this burden to the users or schedulers if they intend to use it. To use a metacomputing system, users usually take the workflow instance as a unit to consider their computation. In other words, they need a distinct namespace for each instance, especially when these instances are executed concurrently. In this scenario, sharing the jobs' name and resolving the conflicts between the names of data files are required. Furthermore, if one instance shares some information with other previously completed instance, only di f f(the different parts) needs to be stored for this instance by taking the previous instance as the base. Although traditional DFSs have some approaches to the above requirements (e.g., we can use directories to simulate the different namespaces, use symbolic link to share a file between different namespaces, and use the tree structure to achieve the di f f mechanism), these approaches are neither elegant nor effective due to the imposed semantics on the directories, preventing them from being modified or moved freely. Furthermore, using symbolic links to share a file is not effective, especially when two files reside at different computational sites.