A Theoretical Study of Anomaly Detection in Big Data Distributed Static and Stream Analytics

摘要

A high volume of data including log records, sensors, monitoring systems, manufacturing processes, call detail records, blogs, emails, and social media streams are generated around the clock by diverse applications. Thus, as the volume of data is growing rapidly, detecting anomaly from high volume big data becomes a critical and difficult task, due to the theatrical (research) and practical (technical) limitations. This paper aims to investigate anomaly detection and provide global understanding of anomaly concepts in the big data mining perspective. In this paper we demonstrate how existing methods of anomaly detection can be adopted with high volumes of data, specifically providing in depth understanding of the anomaly concept in streaming data. The key contribution of this study is an attempt to answer the following questions: 1) What is the concept of big data and what are big data analytic approaches? 2) What is the relationship between big data and anomaly detection? 3) What is the main characteristic of anomaly in big batch and streaming data? 4) What is the appropriate state of the art infrastructure to process and detect large-scale batch and streaming data?.