Handling workload skew in a consistent hashing based partitioning implementation

摘要

Consistent hashing is used for distributing the data uniformly over a given set of servers in a topology. However, uniform distribution of the data over a given set of servers does not guarantee a uniform distribution of the workload associated with the data over the set of servers. When the workload is skewed over a small subset of data items the traditional re-partitioning approach used for handling overloads on a partition fails. This paper analyzes the effect of workload skew on a traditional consistent hashing implementation. A novel approach is then proposed that enables the creation of a uniform distribution even in the presence of skew. The proposed approach is then experimentally verified for correctness.