Performance measurement of data flow processing employing software defined architecture

摘要

With the development of information technology, the importance of big data is quickly highlighted. Big data applications show great value to individuals, companies and governments. Recently, researches on the storage and utilization of big data have achieved considerable results. The prosperity of big data applications is a thrust of drawing attention to the system performance such as timeliness, computational and communication resources. Data retransmission caused by the violation of the stringent delay bound may result in the reprocessing of these data, which would have a negative effect on user experience. To fill this gap, a software defined architecture is developed in this work so that the appropriate start point of processing can be found for the data need to be reprocessed. For further improvement of the processing performance, two models are presented to this software defined architecture. In the optimized model, a priority queue is employed to facilitate the processing efficiency. In addition, data flows transmitting through networks exhibit obvious self-similar characteristics. Performance analysis without taking traffic self-similarity into account may lead to unexpected results. In the optimized model, the tightly coupled system makes performance analysis difficult. Therefore, a decomposition approach is employed to divide the coupled system into a group of single server single queue systems. Finally, the developed model is validated through extensive experimental results.