Big data application in functional magnetic resonance imaging using apache spark

摘要

Recently, big data applications have been rapidly expanding into various industries. Healthcare is among those industries that are willing to use big data platforms, and as a result, some large data analytics tools have been adopted in this field. Medical imaging, which is a pillar in diagnostic healthcare, involves a high volume of data collection and processing. A massive number of 3D and 4D images are acquired in different forms and resolutions using a variety of medical imaging modalities. Preprocessing and analysis of imaging data is currently a costly and time-consuming process. However, few big data platforms have been created or modified for medical imaging purposes because of certain restrictions, such as data format. In this paper, we designed, developed and successfully tested a new pipeline for medical imaging data (in particular, functional magnetic resonance imaging - fMRI) using the Big Data Spark / PySpark platform on a single node, which allowed us to read and load imaging data, convert the data to Resilient Distributed Datasets in order to manipulate and perform in-memory data processing in parallel, and convert final results to an imaging format. Additionally, the pipeline provides an option to store the results in other formats, such as data frames. Using this new pipeline, we repeated our previous work, in which we extracted brain networks from fMRI data using template matching and the sum of squared differences (SSD) method. The final results revealed that our Spark (PySpark) based solution improved the performance (in terms of processing time) approximately fourfold when compared with the previous work developed in Python.