面向有效载荷高速数据流的数据处理方法

摘要

针对卫星有效载荷数传数据传输速度快、实时处理难等特点,提出一种面向有效载荷高速数据流的实时数据处理方法.借鉴MapReduce的多线程并行模式,采用hash算法与归并排序算法相结合的方式,提高数据处理吞吐率,实现实时处理;采用基于XTCE (XML telemetry&command exchange)数据模型的参数解析算法,实现通用性.实验结果表明,该方法能够满足有效载荷对数据处理的实时性和正确性的要求.%Aiming at the characteristics of high speed transmitting and difficulty in real-time data processing for payload transmitted data,a method of data processing for high speed data stream of payload was presented.By drawing on multi-threaded parallel mode of MapReduce framework,hash algorithm and merge sort algorithm were combined to improve the throughput rate of data processing and achieve real-time processing.A parameter parsing algorithm based on XTCE (XML telemetry & command exchange) data model was employed to improve generality.Results of the experiments indicate the proposed method can meet data processing requirements of payload on reabtime performance and validity.