Web crawler and back-end for news aggregator system (Noox project)

摘要

The aim of this manuscript is to develop a web crawler, Content Management System (CMS), and Application Programming Interface (API) for a news aggregator system dubbed Noox project. News aggregator system requires a crawler to populate its content; however, different website may have a different page layout. This projects aims to create a back-end for Noox and open source and scalable web crawler that is capable to extract data from different page layout. The scope of this project includes the web crawler, CMS, API, scheduler for web crawler, and implementation of socket server. The development process utilizes PHP as the primary back-end language with Laravel as the web application framework. The PHP back-end hosts the CMS and the API. The API itself will implement Representational State Transfer (REST) as its architectural design. JavaScript Object Notation (JSON) is used as means of the API to communicate with the clients. The API also implements JSON Web Token (JWT) for client authentication purpose. Python programming language is used to develop the web crawler. The web crawler utilizes BeautifulSoup as the web extraction utility. The web crawler can be adapted to extract data from different page layout by utilizing user created configuration file. It can also be configured to export the extracted data into a JSON file or database system. The result of this project satisfies the requirement of the Noox project.