Research of Self-adaptive Web Page Parser based on Templates and Rules

摘要

Web pages parsing is a concerned topic in recent years, how to get rid of human intervention and formulate extraction rules of subject information from a large number of web pages at the fastest and most accurate speed has becoming an important research point in this field. This paper proposes a frame of self-adaptive web page parser based on templates and rules. Firstly, it uses the noise filter algorithm to filter irrelevant nodes and invalid nodes, and then combines the ways of page template and heuristic rule to generate extraction rules, at the same time it can adjust extraction rules dynamically according to external factors through automatic detection mechanism. Using this frame to generate parsers has better self-adaptability, being able to generate extraction rules better, and being able to locate and extract subject information better. The experimental result shows the effectiveness of the parser.