Topic-specific crawling on the Web with concept context graph based on FCA

摘要

Topic-specific crawling is a method which can not crawl all the webpage, but only crawls the web pages which are related to users' interests. The web pages which have high relevancy of the users' interests should be crawled first. The major problem in focused crawling is how to assign proper credits to the unvisited pages the crawling will visit. In this paper, we propose an effective approach using concept context graph based on Formal Concept Analysis to solve this problem. We build a concept lattice with the visited pages, and then use a method of combination of the term to construct our concept context graph based on the upper concept lattice. Our crawler can measure a page's expected relevancy to a given topic and determine the order in which pages should be visited first. An experiment illustrates that the new method is an effective mechanism which have a considerable result.