PN-FINDR: A Parallelized N-FINDR Algorithm with Spark

摘要

N-FINDR is a well-known method for the endmember extraction of hyperspectral image. However, the N-FINDR processing on hyperspectral data is both data-intensive and computing-intensive. On one side, with development of hyperspectral sensors, the magnitude of hyperspectral data grows quickly, and it is a challenging task to efficiently process the massive data volumes; on the other side, the N-FINDR algorithm needs to scan the whole dataset to compute the volume of the simplex. These make the N-FINDR algorithm very time-consuming when dealing with the big dataset. In this paper, a distributed parallel optimization of the N-FINDR algorithm (PN-FINDR) on Spark is presented. In order to accelerate the distributed parallel computation, the broadcast variables are used to reduce transmission data to each computing node, the intermediate data storage structure is designed to reduce the time consumption of shuffle data transmission, and the RDD cache is used for fast iterative calculation. The experiments conducted on real hyperspectral images of different sizes demonstrate significant improvement on the efficiency of massive hyperspectral data processing.