A versatile data-intensive computing platform for information retrieval from big geospatial data

摘要

AbstractThe increasing amount of free and open geospatial data of interest to major societal questions calls for the development of innovative data-intensive computing platforms for the efficient and effective extraction of information from these data. This paper proposes a versatile petabyte-scale platform based on commodity hardware and equipped with open-source software for the operating system, the distributed file system, and the task scheduler for batch processing as well as the containerization of user specific applications. Interactive visualization and processing based on deferred processing are also proposed. The versatility of the proposed platform is illustrated with a series of applications together with their performance metrics.Highlights•The recent sharp increase of free, full, and open satellite imagery is making Earth Observation truly entering the big data era.•Novel platforms are needed for timely retrieving information from Earth Observation imagery at scale.•A versatile platform coping with batch processing of existing scientific workflows as well as interactive visualization and analysis is put forward.•The versatility of the proposed platform is demonstrated on a variety of actual use cases originating from various application domains.