A Full Text Search Engine for Noisy OCR Document Based on Statistical Language Modeling

摘要

There are large amount of OCR (Optical Character Recognition) documents in digital library. In order to achieve more accurate retrieval results from noisy OCR documents, the traditional way is to correct the erroneous OCR words in advance. The main idea is to replace erroneous words with the most similar corrected one. However such correction does not consider context information and the correction rate sometimes is low. therefore much original information in OCR documents will be lost and the recall and precision ratio of OCR document retrieval are degenerated, especially for short documents. In this paper, we propose a novel OCR document retrieval approach based on statistical language models. First, instead of replacing the erroneous words explicitly, we consider the possibilities of each word in a correction candidate list associating with other words in documents locally and con-textually. Then all the words in the candidate list are indexed as inverted index. After that, the relevant documents are retrieved based on statistical language model and ranked according to the word probability. A known item search experiments on TREC5 confusion track data show that our approach is more effective than OCR documents correction method and the query expansion method, which use n-gram text approximate matching.