Machine-Learning directed Article Detection on the Web using DOM and text-based features

摘要

Burgeoning trend of digital advertising have led websites to embody massive irrelevant content such as page navigation, ads, distractions, promotional videos etc‥. on their pages. In contrast to desktop, limited screen space of mobile carrying variety of ineffectual content magnifies the visual complexity of webpage by disrupting focus of user to read useful content. To solve this issue, Reader mode functionality renders a clutter-free version of web page by stripping out all insignificant elements and letting user to read the content they are actually interested in. Existing implementation in our browser involves 2 modules - Article Detection module that runs heuristics to detect whether a web page is article or not, proceeded by Article Extraction module that extracts the main content of the web page by removing all unwanted elements. Our paper focusses on improving the accuracy of article detection logic by proposing a machine-learning based solution that significantly surpasses the present heuristic-based model in terms of accuracy. In order to get rid of false positives on webpages with meaningless content like product descriptions, we extended our previous solution [1] by incorporating backward-elimination heuristics for web page title and extracting strong Boolean predictors. Our approach also solves the problem of detecting reader mode on Google's AMP HTML web pages where main-content is rendered in an iframe. With this model, we were able to achieve a precision of 0.99 and a recall of 0.94, which outperforms the state-of-the-art techniques which are being used by the major Android based web browsers in the market today.