Learning resources in federated environments: a broken link checker based on URL similarity

摘要

In a large federation of learning resource repositories metadata ean become out of date, especially when resources are moved or deleted, frustrating users and making it critical to assure the availability of resources in any learning environment. Regularly checking all of URLs in a federation with thousands of resources negatively impacts content providers causing poor system performance, heavy network traffic and impoliteness. This paper describes a solution that ensures a balance between the number of resources and the number of checking requests performed. It uses a heuristic algorithm to select a group of resources to verify. Depending on the results, the system either stops checking or continues with a new selection of links constructed from URLs that are similar to the broken links in the previous round. To reach the global optimisation, the selection process also randomly picks other resources with a small probability of error.