A Workflow for Network Analysis-Based Structure Discovery in the Assessment Community

摘要

When technology opens up new domains or areas of research, such as human-agent teaming, new challenges in assessments emerge. Assessments may not be as systematically conducted as new measures develop, and the research may not be as firmly grounded in theory since theories in newer domains are still being formulated. As a result, research in these domains can be fragmented. To address these, an empirically-driven network approach that is complementary to the traditional theory-driven approach is proposed. The network approach seeks to discover patterns and structure in the assessment metadata (.e.g., constructs and measures) that can provide starting points and direction for future research. This paper outlines the workflow of the network approach which comprises three steps: (1) Data Preparation; (2) Data Analysis; and (3) Structure Discovery. As most of the work has been on Data Preparation, the paper will focus on the complexities and issues encountered in the first step, and include broad overviews of the subsequent steps. Anticipated use and outcomes of the network approach are also discussed.