SYSTEMS AND METHODS OF HANDLING INTERNET SPIDERS

摘要

Aspects relate to identifying Internet spiders with an approach involving a plurality of instances of one or more URLs, which reference resources available from a first domain. Instances of the URLs are distributed at other Internet domains. Spiders crawling those domains will activate those URL instances, resulting in requests for the resources referenced by the URLs. A generator of a number of requests for the same resource, from a potential multitude of URL instances, can cause the generator to be categorized as a spider. Similarly, a generator of a number of requests for resources identified by different URLs also can be categorized as spider behavior. In some cases, the first domain may not have a browseable site infrastructure with, such that a spider would not readily crawl it by following internal links. The URLs can refer to custom queries created by various users, who can provide the URLs on their pages, such as on social networking sites.