Aggregating Subjective Objective Measures of Web Search Quality using Modified Shimura Technique

摘要

Web Searching is perhaps the second most popular activity on Internet. Millions of users search the web daily for their purpose. But as there are a number of search engines available, there must be some procedure to evaluate them. In this paper, we try to present an effort in this regard. We are making an attempt to get a comprehensive evaluation system for web search results. We are taking into the consideration the "satisfaction" a user gets when presented with search results The feedback of the user is inferred from watching the actions of the user on the search results presented before him in response to his query, rather than by form filling method. This gives an implicit ranking of documents by the user. Then, classical vector space model for information retrieval is used for computing the similarity of documents selected by the user to that of query. The documents from the search results presented in response to a query are represented by term vectors in vector space. The query is also represented by a term vector. The similarity of a document with the query is thus obtained by computing the dot (scalar) product. Sorting the documents in decreasing order of dot products of their term vectors with that of query, gives a new ranking of the documents on the basis of vector space model. Then, Boolean similarity measure is used to compute the similarity of the documents selected by the user to that of the query and thus another ranking of the documents based on the similarity measure is obtained. We propose a simplified version of a well known Boolean similarity measure and use it for our purpose. All the three rankings obtained in the process is then aggregated using Modified Shimura technique of Rank aggregation. The aggregated ranking is then compared with the original ranking given by the search engine. The correlation coefficient thus obtained is averaged for a set of queries. We show our experimental results pertaining to seven public search engines and fifteen queries.