Creating Content for Educational Testing Using a Workflow That Supports Automatic Item Generation

摘要

Testing organizations like ACT Inc. require large numbers of diverse, high-quality items because computer-based tests are now delivered with variable forms at many times during the year using designs where students receive detailed feedback. But the demand for large numbers of test items far out strips the current supply. Test items, as they are currently written, evoke a process that is both time consuming and expensive because each item is written, edited, and reviewed by an SME. AIG is a method that combines content expertise with technological innovation to solve the practical problem of rapidly expanding the item development process. AIG is a method for using models to generate items with the aid of computer technology. The purpose of our study was to describe the workflow in a strategic partnership between researchers at the University of Alberta with content experts at the testing company ACT Inc. The purpose of this workflow is to help transform item and passage development at ACT Inc. from what is currently a manual, labor-intensive, non-scalable process to a specification driven, automated, highly scalable process. ACT Inc. provides the content expertise in the form of SMEs who are responsible for the creative task of identifying, organizing, and evaluating the content needed to create item models. The University of Alberta research team provides the AIG expertise in the form of research outcomes and computer technology required for the generative task of systematically combining large amounts of information in each item model. By merging the outcomes from the content-based creative task in Stages 1, 3, and 4 with the research-based generative task in Stages 2 and 5, automated processes can be used to facilitate and promote a new approach to item development that is characterized by speed, efficiency, quality, and cost-effectiveness. This workflow was recently used to create 54 different items models in mathematics that produced more than 651,000 unique test items across the elementary, junior high, and high school levels. Mathematics was the focal content area in this project presented in our study. But AIG has also been used in other content areas including science, dentistry, non-verbal reasoning, business management, medicine, and nursing to generate high-quality, content-specific test items efficiently and economically. Hence, the workflow we describe in this study can be used by any testing company that requires large numbers of high-quality assessment tasks.