Big data analytics based Web Crawling System and The Method for searching and collecting open source vulnerability analysis target

摘要

The present invention relates to open source software for uploading open source software developed by a software developer to the information store via the web in order to facilitate mutual information sharing, Open source software vulnerability analysis is performed by collecting open source software based on vulnerability definition information and big data analysis processing by constructing vulnerability risk search keyword set in order to selectively collect open source software likely to be vulnerable to software. The collection time and the inspection time for the target can be reduced. It is also possible to collect open source software with a high risk of vulnerability and to generate new vulnerability search keyword sets whenever vulnerability analysis is performed or to update existing vulnerable keyword sets so that vulnerabilities can be inherent due to more specific and concrete keywords This is a big data analysis based web crawling system for search and collection of open source vulnerability analysis that can collect higher open source software. The present invention relates to a vulnerability analysis module capable of analyzing a vulnerability of an open source software, a vulnerability definition specification DB serving as standard information of a vulnerability analysis for a software vulnerability, and an analysis result obtained by the vulnerability analysis module, Vulnerability analysis result information stored in the information DB, Vulnerability analysis result information DB analysis result information Vulnerability definition corresponding to the information information Vulnerability information for mapping the information information Vulnerability information mapping module for grouping and storing the information mapped by the vulnerability mapping information module Vulnerability analysis engine including information DB; A big data analysis module and a big data analysis module that import vulnerability information mapping information from the vulnerability information mapping information DB of the vulnerability analysis engine and input external vulnerability trend information and define vulnerability risk keywords through keyword frequency number and keyword association analysis The vulnerability is found in the open source software stored in the web-based open source information repository using the vulnerability search data set generation module and the generated vulnerability risk data set that constitute and generate the vulnerable risk keyword Vulnerability to generate a search query to search for open source software that is likely to be open, a search query query module, a vulnerability to collect and store the corresponding open source software through query words generated by the vulnerability search query query module It is characterized by a vulnerability that includes an open source collection module, an open source collection engine, and a generated vulnerability search data set consisting of operating system names and vulnerable keywords.