An Effective Tokenization Algorithm for Information Retrieval Systems

摘要

In the web, amount of operational data has been increasing exponentially from past fewdecades, the expectations of data-user is changing proportionally as well. The data-user expectsmore deep, exact, and detailed results. Retrieval of relevant results is always affected by thepattern, how they are stored/ indexed. There are various techniques are designed to indexed thedocuments, which is done on the token’s identified with in documents. Tokenization process,primarily effective is to identifying the token and their count. In this paper, we have proposed aneffective tokenization approach which is based on training vector and result shows thatefficiency/ effectiveness of proposed algorithm.Tokenization of a given documents helps tosatisfy user’s information need more precisely and reduced search sharply, is believed to be apart of information retrieval. Tokenization involves pre-processing of documents and generatesits respective tokens which is the basis of these tokens probabilistic IR generate its scoring andgives reduced search space. No of Token generated is the parameters used for result analysis.