Distributed data structure templates for data-intensive remote sensing applications

摘要

The remotely sensed images continuously acquired by satellite and airborne sensors are increasing dramatically. Remote sensing applications are overwhelmed with tons of remote sensing data with complex data structures. Efficient programming in parallel systems for data-intensive applications like massive remote sensing data processing will be a challenge. We propose a generic data-structure oriented programming template to support massive remote sensing data processing in high-performance clusters. These templates provide distributed abstractions for large remote sensing image data with complex data structure and allow these distributed data to be accessed as a global one. Through data serialization and one-sided message passing primitives provided by message passing interface, the distributed remote sensing data template whose sliced data blocks are scattered among nodes could offer a simple and effective way to distribute and communicate massive remote sensing data. Efficient parallel input/output directly to and from the distributed data structure will also be offered to address the input/output bottleneck caused by massive image data. Developers can take the advantage of our templates to program efficient parallel remote sensing algorithms without dealing with data slicing and communication through low-level message passing interface APIs. Through experiments on remote sensing applications, we confirmed that our templates were productive and efficient.