Web page downloading and classification

摘要

This paper describes the processes of downloading and classifying Web-based articles in online medical journals as a preliminary step to extracting bibliographic data to populate MEDLINE the widely used database of the National Library of Medicine (NLM). The processes are combined to develop an automated system named "Web Page Downloading and Classification". The system downloads the Web pages using Microsoft's Windows Internet API tool called WinInet, and a combination of several Artificial Intelligence (AI) techniques including the Breadth-First search algorithm and the Constraint Satisfaction method. The Breadth-First search algorithm and the Constraint Satisfaction method are then used to traverse the Web page's links, identify these pages as abstract, full text, PDF or image files, recognize and generate the successors of the downloading pages.