Effects of different response formats in measuring educational research literacy

摘要

The use of appropriate response formats in competency testing has been a topic of interest for the last few decades. Especially the comparison of multiple-choice items with free-response items has been widely examined. The present study examines objective and subjective difficulty of those two response formats and furthermore addresses the question of construct dimensionality based on response formats. Test items measuring Educational Research Literacy were presented to 600 university students in Educational Sciences. To eliminate possible distortions from memory effects, stem-equivalent items of both formats were distributed among two test booklets and linked together with anchoring items. Comparing the response formats did not reveal a clear result concerning objective difficulty. Free-response items were in most cases subjectively rated to be more difficult than multiple-choice items. Objective item difficulty was in most cases not related to subjectively rated difficulty, independent of response format. Model comparisons suggested no differing dimensionality when a method factor based on response format was defined additionally. The results show that in the domain of Educational Research Literacy, there is no distinct advantage of one format over the other in terms of difficulty. This and the established unidimensionality suggest that both formats may be used in competency tests in this content domain. (DIPF/Orig.)