Optimized Template Detection and Extraction Algorithm for Web Scraping of Dynamic Web Pages

摘要

There are a huge number of dynamic websites spanning across the World Wide Web and containing manifold text information that may be helpful for data analysis. Although these dynamic websites generates thousands to millions of web pages for a certain search criteria by user, however, still they represent few unique web templates. Thus identifying the underlying web templates may result in efficient and optimized extraction of information from these web pages. Different techniques such as text_MDL, Minhash jaccard coefficient and Minhash dice coefficient have been used for web template detection and extraction. These path based techniques are complex in nature and not feasible when the number of web documents are very large. A feature based web data extraction algorithm is presented that can identify the underlying template of the web pages represented as DOM tree by clustering the similar web pages together based on feature similarity of web pages and comparing other dynamic web pages with the identified template. It enables processing and extraction of large data sets from heterogeneous web pages in reliable time. We have applied the proposed algorithm on live dynamic web pages of patent portal to compare the result against existing web extraction algorithms and found to be more efficient in terms of throughput.