MESSAGE FAULT TOLERANCE METHOD AND SYSTEM BASED ON SPARK STREAMING COMPUTING FRAMEWORK

摘要

A message fault tolerance method and system based on a Spark streaming computing framework. The method is as follows: creating a workflow program, and configuring a metadata check point task to store streaming computing information, to which a program and configuration version tag is added, in a storage system; configuring Spark streaming to read messages from a message queue to acquire a data set, completing a data set conversion operation in the Spark streaming, and creating a logic execution plan; when state conversion is included in the Spark streaming and a data check point is included in the Spark streaming, configuring the data set to be subjected to processing with state conversion, and adding the program and configuration version tag to a data check point task; when a system has a fault, configuring the Spark streaming to acquire program and configuration version information from a check point, and when the program and configuration version information changes, starting a redo task to redo a check point, and when same does not change, restarting the program; and starting a new version program, and performing subsequent data recovery processing by means of a new version check point.