Deep Learning in the Domain of Near-Duplicate Document Detection

摘要

Increasing of web users due to the popularity of the internet increases the digital documents on the web, and among them many are duplicates and near-duplicates. Identifying duplicate and near-duplicate documents in a huge collection is a significant problem with widespread application and hence, detection and elimination of those documents are the need of the day. This paper proposes a technique to detect the near-duplicate documents on the web which has four main aspects: the first aspect is related to the selection of the important terms from a corpus of documents by developing a new correlation-based feature selection (CBFS) mechanism which enhance the performance of the classifier. The second aspect is to compute the similarity scores between each pair of documents of the corpus. The third aspect concerns with combining these similarity scores with the class label of each pair of documents to generate the feature vector for training the Multi-layer ELM (deep learning architecture) and other established classifiers and the fourth and final aspect introduces a heuristics method to rank the near-duplicate documents based on their similarity scores. The empirical results on DUC datasets witness the effectiveness of the proposed approach using Multilayer ELM as highly appreciable compared to other state-of-the-art classifiers including the deep learning classifiers.