Analyzing Semantic Structure: Testing Simple Metrics from Conceptual Recurrence Analysis using Language Models

摘要

Team communication analyses can provide insight into critical team processes. However, such methods often rely on time-consuming subjective rater evaluations, or on techniques that need extensive preparation and yield results that may be difficult to interpret. As part of an effort to identify reliable automated techniques that can be used on small datasets, this research explores the use Conceptual Recurrence Analysis (CRA) to detect changes in conceptual structure in simulated data. We discuss several metrics that quantify conceptual alignment and test the sensitivity of these metrics to changes in relational structure among groups of words generated from language models. We show that CRA summary statistics are sensitive to changes in relational structure among terms and other manipulations in ways consistent with expectations, and give insight into the changing structure of word distributions as constraints are relaxed. We conclude that CRA presents a sensitive and customizable framework for evaluating linguistic exchanges.