A Novel Mobile Crawler System Based on Filtering off Non-Modified Pages for Reducing Load on the Network

摘要

The studies in the literature show that about 40% of the current Internet traffic and bandwidth consumption is due to web crawlers that retrieve pages for indexing by the different search engines. This traffic and bandwidth consumption will increase in future due to the exponential growth of the web. This paper addresses the problem of bandwidth consumption by introducing an efficient indexing system based on mobile crawlers. The proposed system employs mobile agents to crawl the pages. These mobile agent based crawlers retrieve the pages, process them, compare their data to filter out pages that are not modified after last crawl, and then compress them before sending them to the search engine for indexing. The experimental results of the proposed system are very encouraging.