Research on a professional web crawler application in public security work

摘要

Web crawler as a tool for searching information plays a crucial role in managing and controlling Internet public opinion. With the widespread popularity of the Internet and the growing factors of unsafe and discord, how to use a professional web crawler to improve the efficiency of public opinion monitoring is an imperative problem. Firstly, we analyse the working mechanism of Nutch in detail, which is one of the most popular professional web crawlers. Then, we show two sets of experiments to contrast Nutch with Baidu in four aspects including retrieval satisfaction, retrieval relevance, retrieval repeatability and retrieval accuracy. Lastly, we propose the regional focus control method of public opinion information based on a professional web crawler.