Application of Machine Learning Model on Streaming Health Data Event in Real-Time to Predict Health Status Using Spark

摘要

In healthcare field, a huge amount of data collected in real-time by IoT systems, remote sensing device and other data collection tools brings new challenges that focus primarily on data size and the fast growth rate of such large data. Applying machine learning model on this voluminous data having varying velocity becomes extremely complex for traditional methods of data mining. To deal with this challenge, Apache Spark, a powerful big data processing tool can be used successfully for streaming data event against machine learning through in-memory and distributed computations. This work aims at developing a real-time health status prediction system with breast cancer use case using spark streaming framework with machine learning especially Decision Tree. The system focus on applying machine learning model on streaming data coming with rapid rate to predict health status based on several input variables. Based on this, the system first preprocesses the dataset and analyzes it to create an offline model for learning system, the model then deployed on system and use it in real-time to predict health status.