Research on big data processing technology based on Hadoop

摘要

In the era of big data, the traditional data storage and computing mode can't meet the growing demand of mass data, and the emergence of Hadoop technology is a good solution to this problem. Hadoop is a software framework for distributed processing of large amounts of data, the core components including HDFS and Map/Reduce, and has the advantages of high reliability, high scalability, high efficiency and high fault tolerance. Now Hadoop has been used by many enterprises to build their own data processing platform. Based on the studies of some basic Hadoop big data processing cases, this paper cards a Hadoop-based big data processing architecture, and specifically optimizes the issues of HDFS for storing small files.