Performance evaluation of HDD and SSD on 10GigE, IPoIB RDMA-IB with Hadoop Cluster Performance Benchmarking System

摘要

Hadoop and Map reduce today are facing huge amounts of data and are moving towards ubiquitous for big data storage and processing. Benchmarking tools available are not capable of analyzing performance of the Hadoop cluster system and are made to either run in a single node system. HiBench is an essential part of Hadoop. It is a comprehensive benchmark suit that consists of a complete deposit of Hadoop applications having micro bench marks & real time applications for benchmarking the performance of Hadoop on the available type of storage device (i.e. HDD and SSD) and machine configuration. This is helpful to optimize the performance and improve the support towards the limitations of Hadoop system. In this paper we will present the Performance evaluation of HDD and SSD with 10GigE, IPoIB & RDMA-IB on Hadoop Cluster Performance Benchmarking System. In addition, we will also demonstrate that the traditional servers and old Cloud systems can be upgraded by software and hardware up gradations to perform at par with the modern technologies to handle these loads, without spending ruthlessly on up gradations or complete changes in the system with the use of Modern storage devices and interconnect networking systems.