Adaptation of Classical Machine Learning Algorithms to Big Data Context: Problems and Challenges : Case Study: Hidden Markov Models Under Spark

摘要

Big Data Analytics presents a great opportunity for scientists and businesses. It changed the methods of managing and analyzing the huge amount of data. To make big data valuable, we often use Machine Learning algorithms. Indeed, these algorithms have shown, in the past, their processing speed, efficiency and accuracy. But today, with the complex characteristics of big data, new problems have emerged and we are facing new challenges when developing and designing a new Machine Learning algorithm for Big Data Analytics. Therefore, it is essential to review the classical algorithms to adapt them to this new context. One of the methods of adaptation is the coupling between new technologies (i.e., distributed computing by GPU, Hadoop, Spark) and the Machine Learning algorithms to reduce the computational cost of data analysis. This paper highlights main challenges of adaptation of Machine Learning algorithms to the Big Data context and describes a novel method to make these algorithms efficient and fast in Big Data processing by taking as a case study the Hidden Markov Models using Spark framework. The results of complexity comparison of classical algorithms and those adapted to the Big Data context using Spark show a great improvement.