基于爬虫的有害网站发现与判别系统的实现

摘要

There is a huge number of pornographic, violent, reactionary and other harmful information on the Internet. In order to proactively discover and identify the website containing harmful information, we have implementedaharmful website discovery and identification system based on web crawler. One the one hand, the web crawler proactively discovers the harmful websites, the content security filtering technology determines the legality of the websites, and at last the most likely harmful websites are recommended to us. One the other hand the identification technology can identify the legality of a given website, and shows the judgment reasons to us. The study provides efficient methods of rapidly discovery, automatic recommendation and experts’ confirmation for harmful websites. It will provide a good and trusted network environment to network users.%　　互联网上充斥着大量的色情、暴力、反动等有害信息，为了能够主动发现和判别包含这些有害信息的网站，文章实现了一个基于网络爬虫的有害网站的发现和判别系统。该系统可以通过网络爬虫技术主动去发现有害网站，并通过内容安全过滤技术来判别网站的合法性，把有害网站名单告知给用户。同时，可以根据给定的网站URL判断该网站的合法性，并把判断理由展示给用户。文章提出了一个快速发现、自动推荐、专家确认有害网站的有效解决方案，该系统的应用将会给网络用户提供一个良好可信的网络环境。