Design and implementation of the network video data acquisition system

摘要

Big data technology plays a role in promoting the development of Internet video, and data has become an important carrier for users and network video platform to dig the effect of video communication, guide the video production, and master the user's behavior habits. This system establishes the conceptual process from data acquisition, data processing, data visualization. This article makes analysis and research from the following aspects: the URL extraction module, data acquisition module, data processing module, data visualization module and other functions. For the front page, the system adopts HTML+CSS+JavaScript, for data acquisition core module, the system uses DOM to operate the HTML page node to complete data crawling, and uses MVC architecture of CodeIgniter combined with Ajax technology to achieve the separation between the logical layer data acquisition program and application layer access page . For data visualization, Highcharts is used to display the data so that the data has a better interactivity and presentation capabilities.