Storage tier-aware replicative data reorganization with prioritization for efficient workload processing

摘要

AbstractThe importance of data collection, processing, and analysis is rapidly growing. Big Data technologies are in high demand in many fields, including bio-informatics, hydrometeorology, and high energy physics. One of the most popular computational paradigms used in large data processing frameworks is the MapReduce programming model. Today, majority of integrated optimization mechanisms that quickly produce simple solutions typically consider only load balancing, which is not sufficient for advanced computations. Thus, more efficient and complex approaches are required. In this paper, we suggest an improved algorithm based on categories for reorganizing data in MapReduce frameworks and using replication as well as network transfer. Moreover, we introduce an algorithm customization for urgent computations which require specific approaches in terms of execution time and reliability. We also consider modern data storage aspects, like the ability to work with data on different “layers” (HDD, SSD, and RAM), which can greatly improve the overall performance of our solution.Highlights•A conceptual overview of basic criteria of data placement optimization problem was made.•A heuristic CRUSH algorithm was adopted to the problem conditions and compared with developed own Greedy Algorithm.•Tiering Greedy Algorithm was developed to optimize file location within storage tiering (HDD, SDD, RAM) and was compared with other algorithms.•Categorical Genetic Algorithm (CGA) was developed on base of previously developed Genetic Algorithm for data reorganization•Tiering Categorical Genetic Algorithm was developed and experimentally studied (including parameter sensitivity analysis).