Big Data Processing with Apache Spark in Tertiary Institutions: Spark Streaming

摘要

In tertiary institutions, different set of information are derived from the various department and other functional sections. Individual departments and other functional sections in the institutions manage their data separately. This situation has resulted in huge number of different set of data across the various departments in tertiary institutions. There is no centralized data centre where data/information can be retrieved for the management committee when the need arises. In academic institution data captured is restricted to the institution which collected it but centralisation of the various data in the various functional sections does not exist. This makes it difficult for the management committee to take decisions based on relevant information needed. In order to address this problem, we proposed Spark Streaming. Spark Streaming is an element which facilitates processing of live flows of data. Spark streaming will able to capture data in real time, process it and make it available to the management committee when the need arises