Analysis and Design of Improved Intelligent Search Strategy for Web Crawler

摘要

This paper mainly studies the design and implementation of the search engine's searcher Spider program, and introduces the concept and technical essentials of Spider program in detail. Network crawler is a web crawler program which can run in the background with configuration file as the initial URL crawling down with the width first algorithm and saving the target URL. The paper presents analysis and design of improved intelligent search strategy for Web crawler. Based on multi-thread web crawler, the client can access the server through socket, and the client sends its own set request to the server.