N-Gram: a Method of Conflating Terms An Approach to Text Categorization and Question Answering Systems in the Arabic language

摘要

Our main application program walks through the implementation of theN-Gram technique for Question Answering Systems. The goal of this program is to try to find a paragraph in an Arabic document that can serve as an answer to a question. The implementation uses the Prolog Language. The overall idea is coupling an information retrieval system with a shallow approach to natural language processing. The essential first step in accomplishing this task is the categorization of texts. We mean that for search purposes the search must be guided toward only the related categories: say science, medicine, social problems? Society? history, and other vital categories. Our paper proceeds to attack this vital step, which must be handled as a separate task. We know that this task is already completed in a typical English corpus, such as, for example, the TREC-8 context. We describe the categorization of documents in detail and we also give an overview of advanced topics in this domain. The user asks a question in unstructured language but with a careful choice of words, since document categorization is based on word occurrence information. To process the user's question we use mainly the N-gram, but to enhance the process for high occurrences success we remove some known suffixes, numbers, English words, and others, which are called Stop-Words. This process of removing words is called normalizatioa For simplicity we assumed that the collection of targeted documents is identified ahead of time r and stored as a text file. The rest of the words forming the question are farther processed by the body of our program, which sues N-Grams to compute the similarity between a word and other words from a paragraph of a selected document. Based on the similarity results, we may assign a value. Depending on the values for each word, a selected paragraph may be returned as an answer.