Construction of Search Engine System Based on Multithread Distributed Web Crawler

摘要

The search engine is mainly to the user request information carries on the automatic information collection. Web crawler is an important part of search engine, it is an automatic extraction of web pages for search engines to download web pages from the web. The search system faces hundreds of millions of pages across the Internet, and there are several search servers in each data center, and there may be several crawlers deployed on each grab server. This constitutes a multithread distributed web crawler search system. The paper presents construction of search engine system based on multithread distributed web crawler.