Archiving System of Rural Land Contractual Management Right Data using Multithreading and Distributed Storage Technology

摘要

Due to the fact that volume of the national rural land contract management right data is large, the data are in different formats, the data types are many, and each single file is small, it introduces great challenges to the data transmission in distributed storage system. Data transmission rate is extremely slow, as most of data is small. This article aims to solve the data transmission problem when the rural land contract management right data is archived to the distributed storage system, and to further improve the efficiency of data backup. Firstly, according to the characteristics of rural land contract management right data, this article designs the framework model of distributed storage system, to archive the unstructured original data files. Secondly, based on a single-threaded tool RoboCopy, which is archiving tool and can be used to preserve a subset of data on target systems, this paper uses multi thread technology and design an archiving algorithm. The archiving algorithm assigns each copy task to multiple threads or multiple RoboCopy functions, so that multiple threads are processed concurrently and independently. Although the origin single thread copy ability is better that each thread in multiple threads in the archiving process, but the overall multi-threaded concurrent processing is superior to the origin single thread. Thirdly, using the multiple threaded archiving algorithm, this article design and implement an archiving system for administrator to archive the rural land contract management rights data. The archiving system contains a control flow for administrator to select administrative division, select data list, fill in ID, upload data list, select source data path, add notes, upload data, check consistence, and preserve the data to storage system. Lastly, this article uses and distributed storage system, presents a multi thread transmission algorithm, designs and implements an archiving system based on multithread technology. Finally, this paper takes the real rural land contract management right in Liuhe District of Jiangsu Province as an experiment dataset, and examines the overall effectiveness and usability of the archiving system. The experiment results demonstrate that multithreading technology can reduce the time of data transmission, and improve the efficiency of archiving process effectively; by increasing the number of threads, the efficiency of the data backup system reaches a stable optimal value in a certain number of threads, and although the number of threads is increased indefinitely, the performance of the archiving system will no longer improve. The archiving system designed in this paper can well solve the problem of storage of rural land contract management right data, which provides reference experience and assistance for similar research and application.