Dimensionality of responses to a reading comprehension assessment and its implications to scoring test takers on their reading proficiency.

摘要

This dissertation investigated dimensionality of responses to questions in a passage-based reading comprehension assessment and its impacts on decisions that will be made about test takers' reading proficiency based on their test scores. Test takers' responses to the reading paper of Certificate in Advanced English (CAE) were analyzed using two quantitative approaches -- i.e., confirmatory factor analyses (CFA) and item response theory (IRT) analyses. The following three research questions were addressed in the study. 1. To what extent do items nested within passages exhibit multidimensionality? 2. How does the multidimensionality, if found, affect item parameter estimates when different measurement models are applied to the data? More specifically, how much and in what ways will item parameter estimates be biased when a unidimensional IRT model is applied to multidimensional data? 3. To what extent are person ability estimates influenced by whether multidimensionality is taken into account in a measurement model applied? More specifically, are the ability estimates comparable when estimated from two measurement models that does and does not take into account multidimensionality, respectively?;In order to answer the first research question, four CFA models --- i.e., one-factor, second-order factor, bi-factor and correlated-uniqueness models --- were specified to operationalize claims about the factor structure of test takers' responses and then tested on the basis of statistical fit as well as in terms of their substantive interpretability. The CFA results indicated that the one-factor model does not fit to the data and also that the bifactor model is the model that can best represent inter-item relationships on the CAE reading paper. These results imply that the item responses on the CAE reading test are multidimensional and that at least two sources of item inter-correlations --- reading proficiency and the specific reading passage on which items are based --- should be taken into account in order to properly model the item responses on the test.;The second and third research questions addressed the consequences of applying different measurement models that does or does not take into account multidimensionality observed in previous CFA analyses. To answer these research questions, two IRT models -- unidimensional and bifactor IRT models --- were applied to the data and the results were compared in terms of item parameter and person ability estimates. The results showed that an IRT model that takes into account conditional dependence among nested items within a reading passage, namely the bifactor model, provided different item parameter estimates from a model that assumes unidimensionality. These differences were particularly pronounced for the item discrimination parameter estimation. Furthermore, the use of different IRT models resulted in differences in scaling test takers' abilities. The finding that test takers were scored differently in different measurement models is particularly noteworthy, when it comes to the use of test scores for making decisions about individual test takers.;Based on the aforementioned results, this dissertation stresses that dimensionality of test takers' responses should be taken into account when choosing the most appropriate measurement model to be applied. The dissertation concludes with implications for using language assessment results in the real world -- making use of test scores for an intended purpose in both meaningful and equitable ways.