分布式智能垂直搜索引擎系统的设计

摘要

This system aims to develop distributed intelligent vertical search engine system in network environment, giving priority to web page is the original object which it deals with, and they are scattered on different nodes in the whole network diagram. In order to improve the intelligent and automated platform, we must take the platform as entirety to develop, achieve data' s automatic collection (web crawler), information extraction (web resolution), efficient processing (index, storage, clustering), effective use(storage, clustering), and ultimately form a unified system, realize and improve the needed basic environment(distributed storage and parallel computing) which these problems are solved.%本系统旨在开发一个网络环境下的分布式智能垂直搜索引擎系统,其所处理的原始对象以网页为主,它们分散在整个网络图的不同节点上,为提高平台的智能化、自动化,必须把平台作为一个整体进行开发,实现数据的自动采集(网络爬虫)、数据的信息抽取问题(网页解析),数据的高效加工(索引、存储、聚类),数据的有效利用(检索、聚类),最终形成一个统一的系统,实现并完善为解决这些问题所需要的基础环境(分布式存储与并行计算).