Data-Intensive Text Processing with MapReduce Jimmy Lin and Chris Dyer

摘要

The world has been blessed by the ever-growing World Wide Web and the asso- ciated vast amount of information available through commercial search engines. Many subfields of computational linguistics, such as speech recognition, machine translation, summarization, coreference resolution, question answering, word sense disambiguation, and so on, are playing increasingly important roles in information extraction from the Web. At the same time, the Web itself is also providing more and more data for computational linguists to study. For example, in the past decade, the amount of text available for speech recognition and machine translation research has increased by several orders of magnitude. Corpora consisting of hundreds of millions of words are not uncommon anymore. This clearly poses serious challenges in terms of computation for traditional text processing approaches using a single computer. As a result, efficient distributed computing has become more crucial than ever.