Directed web crawler with machine learning

摘要

A web crawler identifies and characterizes an expression of a topic of general interest (such as cryptography) entered and generates an affinity set which comprises a set of related words. This affinity set is related to the expression of a topic of general interest. Using a common search engine, seed documents are found. The seed documents along with the affinity set and other search data will provide training to a classifier to create classifier output for the web crawler to search the web based on multiple criteria, including a content-based rating provided by the trained classifier. The web crawler can perform it's search topic focused, rather than “link” focused. The found relevant content will be ranked and results displayed or saved for a specialty search.