New techniques of high-speed data processing for spacecraft observation data via NICT Science Cloud

摘要

A variety of satellite missions, Earth observations and environments, Space physics and Astrophysics and Commercial uses, are carried out every year. Most of the satellites yield a large-scale data, and high-performance data processing technologies are expected. We have been developing a cloud system (the NICT Science Cloud) for big data analyses of Earth and Space observations via spacecraft. In the present study we propose a new technique to process large-scale data with paying attention to the fact that high-speed I/O (data file read and write) is important compared with data processing itself. We also adopt a task scheduler, the Pwrake, for easy management of parallel data processing. Using a long-time scientific satellite observation data (GEOTAIL satellite), we examine performance of the system on the NICT Science Cloud. We successfully archive high-speed data processing more than 100 times faster than those on traditional data processing environments. Since the design of the present system is for versatile use, we also discuss how to apply for other satellite missions such as GOSAT satellite, KAGUYA satellite and ISOSIM-L (a radar simulator).