An Approach to Extracting Central URLs on Catalog Page

摘要

Catalog pages construct the intermediate layer in architecture of a standard web site; therefore research on information retrieval for this kind of pages can be beneficial to improve web crawler's efficiency. A page is called "Catalog-style" if its main body is displayed as a sequence of regular entries, and the central link in each entry apparently contains the page’s major information. Here, we propose a central-URL extraction approach, which can automatically recognize effective information from the main segmentation on catalog-page. Our approach combines machine learning classification and DOM (Document Object Model) tree based analysis. For one page, we represent each block node, mainly DIV and Table, by a set of content-based and structure-based features, which can be used as the input of support vector machine to determine whether it belongs to "Main-Body" or not. After identifying the main semantic block, a DOM tree based algorithm that utilizes catalog's heuristic rules is implemented to find the central URLs in the segmentation. The evaluation results show that our approach obtains encouraging results with a high recall/precision ratio. This can be applied in topic-specific search engine development and other Web applications.