TASK-CRAWLING SYSTEM AND TASK-CRAWLING METHOD FOR DISTRIBUTED CRAWLER SYSTEM

摘要

A task-crawling system for a distributed crawler system includes a controlling end, a crawling end, and a task monitoring module. The crawling end acquires a corresponding task, and sends data of the task to the controlling end. The controlling end works for assigning a number to the task, defining a timeout period for the task, generating a task-distribution event, and storing timestamp data of distribution of the task. The controlling end distributes the task distribution to the task monitoring module and the crawling end. The crawling end performs corresponding crawling logic to the crawl task, and sends information about completion of the task to the controlling end. In case of abnormality that prevents the crawl task from being performed properly, the task monitoring module re-pushes the task to the controlling end, thereby avoiding failure of the task otherwise caused by web-related problems.