AgentMat: Framework for data scraping and semantization

摘要

Most of the enormous amount of information from the internet is available just like Web pages made for a human reader. They don't have any common interface for accessing, searching or browsing the data. Hence, it's hard to extract the semantic data from the Web, categorize them and keep them updated. For this purpose we have designed and implemented a system called AgentMat. This system is designed for efficient extraction of large amount of data from the Web pages. AgentMat processing is based on an XML-based language describing the given extraction task in a declarative way. The task description consists of system components, which connected together are able to perform the desired functionality on a general Web page. Thanks to this scraping system the raw contents from the irregularly updated and unstructured Web pages can be kept categorized and accessed together with the semantic metadata. In our pilot implementation we have built the MediaPub system, which extracts the information from various Web pages, does automatic categorizing and checks for duplicities.