From a Smoking Gun to Spent Fuel: Principled Subsampling Methods for Building Big Language Data Corpora from Monitor Corpora

摘要

With the influence of Big Data culture on qualitative data collection, acquisition, andprocessing, it is becoming increasingly important that social scientists understand the complexityunderlying data collection and the resulting models and analyses. Systematic approaches for creatingcomputationally tractable models need to be employed in order to create representative, specializedreference corpora subsampled from Big Language Data sources. Even more importantly, any suchmethod must be tested and vetted for its reproducibility and consistency in generating a representativemodel of a particular population in question. This article considers and tests one such method forBig Language Data downsampling of digitally accessible language data to determine both howto operationalize this form of corpus model creation, as well as testing whether the method isreproducible. Using the U.S. Nuclear Regulatory Commission’s public documentation databaseas a test source, the sampling method’s procedure was evaluated to assess variation in the rate ofwhich documents were deemed fit for inclusion or exclusion from the corpus across four iterations.After performing multiple sampling iterations, the approach pioneered by the Tobacco DocumentsCorpus creators was deemed to be reproducible and valid using a two-proportion z-test at a 99%confidence interval at each stage of the evaluation process–leading to a final mean rejection ratioof 23.5875 and variance of 0.891 for the documents sampled and evaluated for inclusion into thefinal text-based model. The findings of this study indicate that such a principled sampling method isviable, thus necessitating the need for an approach for creating language-based models that accountfor extralinguistic factors and linguistic characteristics of documents.