Using the ECD Framework to Support Evidentiary Reasoning in the Context of a Simulation Study for Detecting Learner Differences in Epistemic Games

摘要

The evidence-centered design (ECD) framework is a powerful tool that supports careful and critical thinking aboutthe identification and accumulation of evidence in assessment contexts. In this paper, we demonstrate how the ECDframework provides critical support for designing simulation studies to investigate statistical methods within an ill-definedmethodological domain like games-based assessment. We discuss the design and selected findings of acomplex simulation study to investigate the utility of statistics derived from a non-parametric method calledepistemic network analysis; this method is used in practice for the analysis of real data from a suite of digitallearning environments called epistemic games. We present findings that show that one marginal ENA statistic, theweighted density statistic from social network analysis, has some ability to detect differences among a variety ofsimulated learners when they play different types of simulated games. Detection strength ranged from weak tostrong, depending jointly on game design characteristics and the types of learners being compared. Our workillustrates the complex challenges of how best to describe, justify, and evaluate design decisions for simulationstudies in the context of games-based assessment.