Query Triggered Crawling Strategy: Build a Time Sensitive Vertical Search Engine

摘要

In today's information society, it is important to retrieve fresh information [1]. Many of the vertical search results are valid to users in only a short period of time. But due to resource constraints, it is not possible to keep the entire local storage synchronized with the Web [2]. We implemented a TSVS (Time Sensitive Vertical Search Engine) prototype named Velocisaurus focused on time-critical airfare discount information search to investigate the time critical requirements of vertical search and proposed a QTC (query triggered crawling) strategy to coordinate the crawling systems by real-time user queries and solve this problem. Experiment shows that QTC driven crawlers significantly improves the freshness of the search results and utilizes the resources more efficiently compared to regular search engine crawlers.