Adaptive cache policy scheduling for big data applications on distributed tiered storage system

摘要

Multitiered storage systems, which are made up of heterogeneous devices, are widely usedin distributed environments to accelerate the I/O performance of upper big data applications.It raises new challenges in efficient data migration through smart caching mechanisms amongheterogeneous storage levels, such asMEM-SSD-HDD. To optimize the cache policy schedulingmechanism on the distributed tiered storage architecture, we proposed a general frameworkwith five layers, including a tiered storage system layer, a cache migration policy layer, a cachepolicy adaptive scheduling layer, a data access pattern layer, and a big data application layer. Theframework prototype has been designed and implemented on the widely used distributed hybridstorage system named Alluxio. To meet the demands of the big data application layer, on the onehand, we designed a couple of cache eviction policies and promotion policies covering variousaccess patterns on the cache migration policy layer (several proposed eviction policies havebeen adopted by the Alluxio open-source community). On the other hand, two adaptive cachepolicy scheduling algorithms for selecting appropriate policies in various scenarios are designedand implemented on the cache policy adaptive scheduling layer. The scheduling algorithms aredesigned based on the hit ratio statistics and data access pattern model prediction, respectively.Experimental results show that the proposed cache policies are very effective for various bigdata applications, such as Spark SQL. The proposed cache policy scheduling algorithms withvarious eviction policies can improve around 20% hit ratio than that with a single eviction policy.