Automatically Building a Corpus for a Minority Language from the Web

摘要

We present an approach to language-specific query-based sampling which, given a single document in a target language, can find many more examples of documents in that language, by automatically constructing queries to access such documents on the world wide web. We propose a number of methods for building search queries to quickly obtain documents in the target language. They perform accurately and efficiently for building a corpus of documents in Tagalog starting from a single seed document, when these documents are only 2.5% of the documents in a collection. We found that a simple approach - of sampling with a query consisting of the most frequent word from the minority language corpus constructed so far - was very successful. This method built a corpus of documents with word frequencies similar to those in the corpus based on all Tagalog documents in our collection, and required a relatively small number of search queries. It also quickly acquired a good coverage of vocabulary terms. However, adding an element of randomness to the query may give greater coverage, although more queries are required.