基于统计的网页净化模板生成算法

摘要

A web site' s page purification template generation algorithm is proposed based on statistics. At first the DOM trees of a group of pretreated sample pages in a certain web site are extracted. For each DOM tree, the node with the most number of words in the sibling will be retained and form an unilateral subtree. Merging these subtrees, the highest number of node in the sibling is the important content node, and these important nodes form another unilateral subtree. In the subtree, if a child node has less than a half words of its parent node, its parent node will be the last one having the important content.%同一个站点的大部分网页拥有几乎相同的DOM标签树,处理后的标签树作为一个模板,该站点的所有网页只保留这个模板中叶子节点包含的内容,由此可以实现这个站点的所有网页的净化.首先对一个站点内的一组样本网页提取内容块树,针对每个树统计每个标签节点包含的文本字数,同级节点只保留字数最多的一个,从而生成单边子树UST；然后把这组UST合并,同级节点中出现次数最多的即为重要内容节点,把这些节点串起来就构成重要单边子树PUST;最后比对每个父节点与子节点之间的字数,当比值超过一个阈值时则删除子节点以下的所有节点,从而生成该站点的重要单边子树SPUST.这个SPUST就是该站点的网页净化模板.