Distributed system for downloading internet information and a method thereof

摘要

A distributed system for downloading distributed Internet information and a method thereof are disclosed. A distribution subsystem receives downloading task information from a user, and sends the downloading task information to a downloading subsystem. The downloading subsystem downloads task result information according to the downloading task information, and sends the task result information to the distribution subsystem. The distribution subsystem receives the task result information, and returns the task result information to the user made the request. The physical environment of the system can include a distribution server and a downloading subsystem group, the internal function of the system is to assign the required tasks to the downloading subsystem group based on a queue model, the downloading subsystem group can process the assigned tasks concurrently. The system and the method is adapt to a mass-data acquisition over Internet, which may significantly improve the efficiency and timeliness for obtaining information.