Design of a Parallel and Scalable Crawler for the Hidden Web

摘要

The WWW contains a huge amount of information from different areas. This information may be present virtually in the form of web pages, media, articles (research journals/magazine), blogs, etc. A major portion of the information is present in web databases that can be retrieved by raising queries at the interface offered by the specific database and is thus called the hidden web. An important issue is to efficiently retrieve and provide access to this enormous amount of information through crawling. In this paper, the authors present the architecture of a parallel crawler for the hidden web that avoids download overlaps by following a domain-specific approach. The experimental results further show that the proposed parallel hidden web crawler (PSHWC) effectively and efficiently extracts and downloads the contents in the hidden web databases.