How should one perform matching in observational studies when the units aretext documents? The lack of randomized assignment of documents into treatmentand control groups may lead to systematic differences between groups onhigh-dimensional and latent features of text such as topical content andsentiment. Standard balance metrics, used to measure the quality of a matchingmethod, fail in this setting. We decompose text matching methods into twoparts: (1) a text representation, and (2) a distance metric, and present aframework for measuring the quality of text matches experimentally using humansubjects. We consider 28 potential methods, and find that representing text asterm vectors and matching on cosine distance significantly outperformalternative representations and distance metrics. We apply our chosen method toa substantive debate in the study of media bias using a novel data set of frontpage news articles from thirteen news sources. Media bias is composed of topicselection bias and presentation bias; using our matching method to control fortopic selection, we find that both components contribute significantly to mediabias, though some news sources rely on one component more than the other.
展开▼
