Load rebalancing for Hadoop Distributed File System using distributed hash table

摘要

All the machines are required to be under a common administrator and be able to communicate securely. To communicate securely, Advanced Encryption Standard (AES) algorithm is used to enable protection of data at each clusters. It performs encryption and decryption before read and write respectively. Encryption and Decryption in key is used for securing Hadoop Distributed File System. The existing system depends on a single name node to manage almost all operations of every data block in the file system. As a result it can be a bottleneck resource and a single point of failure. To overcome this failure, Load Rebalancing Algorithm is used for Hadoop Distributed File System using Distributed Hash Table. The Proposed load rebalancing algorithm will be compared against a centralized approach in a production system and a competing distributed solution is presented in the literature. The storage nodes are structured as a network based on distributed hash table discovering a file chunk can simply refer to rapid key lookup in DHTs, given that a unique handle (or identifier) is assigned to each file chunk.