MapReduce Hadoop Models for Distributed Neural Network Processing of Big Data Using Cloud Services

摘要

The paper proposes a formalization process of Big Data distributed intelligent processing using Cloud-Fog-Dew architecture. This process provides specialized services, based on continuous support of experts in areas of concern, and advisory support for their actions in diagnostically complex cases. The method of Big Data processing based on neural networks is considered, which is distinguished by dynamic redistribution of work between computers, that allows to uniformly load the computing cluster with different data topologies. Proposed method is less by one order of computational complexity and less time spent. The MapReduce Hadoop models for distributed neural network processing of Big Data were proposed, characterized by the adaptation of data topology to the corresponding architectural computer cluster. This reduces the amount of information transmitted between nodes to increase productivity in solving complex tasks and effectively balancing the load of computing resources with different data topologies. An experimental Hadoop cluster was created to evaluate the performance of developed models for Big Data distributed processing. It allows for the implementation of parallel learning procedures for multilayer neural networks based on "star" and "fully connected graph" data topology with different amounts of input data.