An efficient distributed caching for accessing small files in HDFS

摘要

AbstractIn this paper, we propose a distributed caching scheme to efficiently access small files in Hadoop distributed file system. The proposed scheme reduces the volume of metadata to manage in the NameNode by combining and storing multiple small files in a block. In addition, it reduces unnecessary accesses by maintaining information on requested files using client cache and DataNode cache, and synchronizing metadata of the client cache. The client cache maintains small files requested by users and metadata, and each DataNode cache maintains small files frequently requested by users. Performance evaluation shows that the proposed distributed cache management scheme significantly outperforms existing schemes in small file access costs.