Jobcast-Parallel and distributed processing framework Data processing on a cloud style KVS database

摘要

In this paper, we propose a new architecture for parallel and distributed processing framework, "Jobcast", which enables data processing on a cloud style KVS database. Nowadays, lots of KVS (as known as Key Value Store) systems exist which achieve high scalability for data spaces among a huge number of computers. Some of KVS implementations use "consistent hash" algorithm to identify the backend data node to store a pair of key and value. Jobcast also uses consistent hash algorithm for a distribution strategy and has a capability to store key and value pairs into huge number of computers as a KVS system. Furthermore, Jobcast also distributes "jobs" into data nodes for parallel and distributed processing. In this paper, we introduce a basic architecture of Jobcast and evaluate a data processing performance for a typical example.