Hadoop based real-time Big Data Architecture for remote sensing Earth Observatory System

摘要

Recently, Big Data analytics emerged as a hot topic because of the incredible growth of the information and communication technology. One of the exceedingly anticipated key contributors of the Big Data is real-time Earth Observatory System (EOS). Although the data generated by the individual satellite in EOS may not be significant, the overall data generated across numerous satellites may yield to the significant amount of the Big Data. Thus, extracting the useful information in an efficient manner leads a system towards major computational challenges in EOS, such as, to analyze, to aggregate, and to store, where data is remotely collected. Therefore, the paper proposes a set of requirements for achieving pervasive, integrated information system of EOS and associated services (real-time and offline data processing). The Big Data Architecture is also proposed to address all the aspect of the Big Data ecosystem and includes the following components: Data Acquisition Unit, Data Processing Unit, Data Storage Unit, and Data Analysis and Decision Unit. The proposed architecture is termed as Holistic as it considers the flow of data from satellites to services, which is designed for efficiently process and analyze the Big Data. Finally, a detailed analysis of remotely sensed earth observatory Big Data for Land and Sea area are provided using UBUNTU 14.04 LTS core™i5 machine with 3.2 GHz processor and 4 GB memory. The results show that the proposed network architecture efficiently process EOS data at a real-time as well as offline.