Distributed Chunk-Based Framework for Parallelization of Sequential Computer Vision Algorithms on Video Big-Data

摘要

In this paper we propose a complete framework that enables big-data tools to execute sequential computer vision algorithms in a scalable and parallel mechanism with limited modifications. Our main objective is to parallelize the processing operation in order to speed up the required processing time. Most of the present big-data processing frameworks distribute the input data randomly across the available processing units to utilize them efficiently and preserve working load fairness. Therefore, the current big-data frameworks are not suitable for processing huge video data content due to the existence of interframe dependency. When processing such sequential computer vision algorithms on big-data tools, splitting the video frames and distributing them on the available cores will not yield the correct output and will lead to inefficient usage of underlying processing resources. Our proposed framework divides the input big-data video files into small chunks that can be processed in parallel without affecting the quality of the resulting output. An intelligent data grouping algorithm was developed to distribute these data chunks among the available processing resources and gather the results out of each chunk using Apache Storm. The proposed framework was evaluated against several computer vision algorithms and achieved a speedup from 2.6x up to 8x based on the algorithm.