Zipfian frequency distributions facilitate word segmentation in context

摘要

Word frequencies in natural language follow a highly skewed Zipfian distribution, but the consequences of this distribution for language acquisition are only beginning to be understood. Typically, learning experiments that are meant to simulate language acquisition use uniform word frequency distributions. We examine the effects of Zipfian distributions using two artificial language paradigms-a standard forced-choice task and a new orthographic segmentation task in which participants click on the boundaries between words in contexts. Our data show that learners can identify word forms robustly across widely varying frequency distributions. In addition, although performance in recognizing individual words is predicted best by their frequency, a Zipfian distribution facilitates word segmentation in context: The presence of high-frequency words creates more chances for learners to apply their knowledge in processing new sentences. We find that computational models that implement " chunking" are more effective than " transition finding" models at reproducing this pattern of performance.