Job starvation avoidance with alleviation of data skewness in Big Data infrastructure

摘要

During the age of rush in the need for big data, Hadoop is a postulate or cloud-based platform that has been heavily encouraged for all solutions in the business world's big data problems. Parallel execution of jobs consists of large data sets is done through map reduce in the hadoop cluster. The completion of job time will depend on the slowest running task in the job. The entire job is extended if one particular job takes longer time to finish and it is done by the delayer. An inequality in the measure of data allocated to each individual task is referred to as Data skewness. An efficient dynamic data splitting approach on Hadoop called the Hybrid scheduler who monitors the samples while running batch jobs and allocates resources to slaves depending on the complexity of data and the time taken for processing. In this paper, the effectiveness of web swarming is showcased using hadoop eliminating Distributed Denial of Service (DDoS) attack detection scenarios in the Web servers. Query processing is done through Map Reduce in traditional Hadoop clusters and is replaced by the proposed Block chain query processing algorithm. Thereby improvise the execution time of the assigned task in the proposed system to mitigate the data skewness. The main aim of this paper is to avoid job starvation thus minimizing the response time efficiently during the process and mitigating data skewness in existing system.