Architectures enabling scalable Internet search

摘要

The vast amount of Internet content becomes manageable mainly by means of search engines that allow users to enter queries into a web form and receive as result a list of matches that refer to Intenet content elements, such as the URLs identifying matching HTML pages. However, the quality of these search engines suffers from two conceptual problems. The content volume grows faster than the bandwidth available to index it, and a large and growing share is ``hidden'' in the {em deep web}, e.g. behind HTML forms, making it hard to reach and index by search engines. The work presented here shows that these problems can be overcome if the paradigm of Internet search is reversed: content providers have to assist in making their content searchable. This leads to a distributed architecture that scales better than the central approach that current search engines implement, and that makes the deep web searchable. A UML model of the distributed search architecture was created and then implemented using Java, verifying the feasibility of the concepts. The scalability of the solution was proven using a formal model of the bandwidth consumed by a specific class of distributed search algorithms, as used by the suggested architecture. The remaining problem of how to create the content so that it complies with the suggested search architecture was tackled in two ways. Adapters for existing content can be created with little effort, as has been shown by a prototype. New Internet applications can be made searchable using the Model-Driven Architecture approach as introduced by the Object Management Group. A metamodel with a corresponding UML profile was defined that allows for a compact specification of an application's searchability. Using model transformations, a large share of the code that implements the specified searchability can be generated automatically from the models expressed in this metamodel.