Observer Agreement for Timed-Event Sequential Data: A Comparison of Time-Based and Event-Based Algorithms

摘要

Observer agreement is often regarded as the sine qua non of observational research. Cohen’s kappa is a widely-used index and is appropriate when discrete entities, such as a turn-of-talk or a demarcated time-interval, are presented to pairs of observers to code. Kappa-like statistics and agreement matrixes are also used for the timed-event sequential data produced when observers first segment and then code events detected in the stream of behavior, noting onset and offset times. Such kappas are of two kinds, time-based and event-based. Available for download is a computer program (OASTES, Observer Agreement for Simulated Timed Event Sequences) that simulates the coding of observers of a stated accuracy, and then computes agreement statistics for two time-based kappas (with and without tolerance) and three event-based kappas (one implemented in The Observer, one in INTERACT, and one in GSEQ). Based on simulation results presented here, and due to the somewhat different information provide by each, reporting of both a time-based and an event-based kappa is recommended.