基于云计算的商业情报采集系统

摘要

The business intelligence gathering system is different from the traditional search engine system. The data classification and clustering techniques of the traditional search engine can not fully meet the special needs of timeliness and pertinence in the business intelligence gathering process. This paper presents a solution to business intelligence gathering,by using Bayesian classification algorithm and deleting duplicated web pages algorithms in the cloud computing environment to achieve internet data's real-time capturing,analysis,classification and clustering,and form the omnibearing and three-dimensional intelligence noumenon of users. The amount of data captured is stored in a distributed file system. The gathered information is stored in the cloud database CouchDB.%商业情报采集系统不同于传统的搜索引擎系统,情报具有时效性、针对性等特点,传统搜索引擎中的数据分类和聚类技术不能完全满足商业情报采集过程中对时效性和针对性的特殊需求.提出一种商业情报采集解决方案,在云计算环境中采用贝叶斯分类算法和多种网页去重、提取等算法,实现对互联网数据的实时性抓取、分析、分类、聚类,形成对用户全方位立体化的情报本体,抓取的海量数据采用分布式文件系统存储,采集的情报用基于云的数据库CouchDB存储.