A Focused Crawler Framework for E-commerce Deep Web Based on Concept Analysis

摘要

With the development of electronic commerce, product information on the Internet is growing rapidly. This paper presents a framework for the e-commerce Deep Web focused crawler. In order to overcome the deficiency of topic filtering strategy based on keywords, an algorithm for computing the degree of correiativity based on concept analysis is proposed. A noise filtering method based on statistics is proposed to overcome the noises in result set pages and greatly reduces the topic drift. Experimental results show that the framework are effective, and the accuracy and efficiency of crawling are both improved.