Focused Crawler Based on Domain Ontology and FCA

摘要

Focused crawler is a web crawler that aims to selectively seeking out web pages which are relevant to a predefined set of crawling topics, instead of searching the whole Web exhaustively. In this paper, we propose an effective focused web crawling method which based on domain ontology and Formal Concept Analysis (FCA). The method construct a core similarity graph based on WordNet and concept relatedness firstly, and then combining with concept lattice knowledge, a Similarity Concept Context Graph (SCCG) is built. On the basis of SCCG, a focused web crawling method which can measure a page's expected relevancy to a given topic and determine which URL should be crawled firstly is proposed. Experimental result shows our approach has higher recall rates than the standard breadth-first approach, the approach with Context Graph (CG) and the approach with Relevancy Context Graph (RCG). In conclusion, the result demonstrates the effectiveness and significance of our approach.