Scalability and Efficiency Challenges inrnLarge-Scale Web Search Engines

摘要

Commercial web search engines need to process thousands ofrnqueries every second and provide responses to user queriesrnwithin a few hundred milliseconds. As a consequence ofrnthese tight performance constraints, search engines constructrnand maintain very large computing infrastructuresrnfor crawling the Web, indexing discovered pages, and processingrnuser queries. The scalability and eu000eciency of theserninfrastructures require careful performance optimizations inrnevery major component of the search engine.rnThis tutorial aims to provide a fairly comprehensivernoverview of the scalability and eu000eciency challenges in largescalernweb search engines. In particular, the tutorial providesrnan in-depth architectural overview of a web search engine,rnmainly focusing on the web crawling, indexing, and queryrnprocessing components. The scalability and eu000eciency issuesrnencountered in these components are presented at four diu000berentrngranularities: at the level of a single computer, a clusterrnof computers, a single data center, and a multi-center searchrnengine. The tutorial also points out some open researchrnproblems and provides recommendations to researchers whornare new to the feld.