Pattern matching for extraction of core contents from news web pages

摘要

Web pages, besides core contents, consist of other elements, such as banners, navigational elements, copyright information, external links, etc. This noisy content covers more area of web pages and is typically not related to the main subjects of the web pages. Most of the information available on web pages is either represented in XML, or HTML, or XHTML format that mostly contains semi-structured text documents, which lacks formatted document structure. This document does not discriminate between the text and the schema, and the amount of structure used to represent the text depends on the purpose. No semantic is applied to semi-structured documents. This requires extracting core contents of text document to analyse words or sentences for retrieving relevant information. Although there are many existing methods that formulate the actual content identification problem as a DOM tree node selection problem, each one has some sort of lacunae. Here we proposed an approach based on pattern matching technique. This technique uses simple heuristic for extraction of core contents from web pages which are mostly semi-structured in nature. It requires visiting the appropriate news web site using their URL, accessing the links related to each news page of specified category, extracting the data including metadata from each of these news web pages. The approach uses devised algorithm that applies regular expressions (regexes) to identify the correct pattern for extracting the actual text contents from these news documents. Proposed approach deals with news web pages of any size and extracts core contents with efficiency and high accuracy.