On one hand, redundant pages could increasesearching burden of the search engine. On the other hand,they would lower the user’s experience. So it is necessary todeal with the pages. To achieve near-replicas detection, mostof the algorithms depend on web page content extractioncurrently. But the cost of content extraction is large and it isdifficult. What’s more, it becomes much harder to extractweb content properly. This paper addresses these issuesthrough the following ways: it gets the definition of thelargest number of common character by taking antisenseconcept of edit distance; it suggests that the feature string ofweb page built by a previous Chinese character of period insimple processing text; and it utilizes the largest number ofcommon character to calculate the overlap factor betweenthe feature strings of web page. As a consequence, thispaper hopes to achieve near-replicas detection in high noiseenvironment, avoiding extracting the content of web page.The algorithm is proven efficient in our experiment testing:the recall rate of web pages reaches 96.7%, and theprecision rate reaches 97.8%.
展开▼
