Design and application of intelligent dynamic crawler for web data mining

摘要

Web data acquisition is the foundation of Web data mining. Web crawler is an important tool for Web data acquisition, but the frequent updates of Web data structures, data sources and distribution channels, resulted in high costs of crawler program development and maintenance. In order to solve this problem, this paper designed and implemented an intelligent dynamic crawler, which stored the data extraction rules of XPath in database, loaded the rules dynamically according to the target, and used TF-IDF method to calculate the relevance. The Web crawling rules can be automatically acquired, which made the crawler intelligent and dynamic, improved the adaptability of the crawler for the complex web environment, and reduced the maintenance and update cost. Finally, this paper applies the intelligent dynamic crawler to the threat awareness of public vulnerabilities, with the method of data collection and analysis of the vulnerability community and the network node search engine. The experiment used the prototype system on three vulnerability communities to collect and analyze the data. The results showed that the intelligent dynamic crawler can realize the high-efficient and flexible data collection function, and laid the foundation for Web data mining.