Enabling a web-crawling robot to collect information from web sites that tailor information content to the capabilities of accessing devices

摘要

A web-crawling robot retrieves information from a web server that tailors information content to the capability of an accessing device. A link deriving unit in a proxy server for relaying data exchanged between the robot and the site analyzes a response from the site to the robot, and acquires information on a user agent corresponding to a particular kind of content of a link destination. On the basis of the information, a user agent information editing unit in the proxy server adds user agent information to the content retrieval request from the web-crawling robot to the site so as to disguise it as a content retrieval request issued from a given user agent, thereby acquiring a response corresponding to capabilities of the user agent.