Impact of the number of peers on a mutual assessment as learner’s performance in a simulated MOOC environment using the IRT model

摘要

We discuss the problem of setting the best number of peers to which a given evaluation job should be assigned, in a Peer Assessment setting. The Peer Assessment is supposed to happen in a large scale class, such as in the case of Massive Open Online Courses. We use a dataset that simulate a large class (1000 students), based on Gaussian distributions of the Student Model features. Such features are related to the student’s proficiency, and assessment capability. The number of peers assigned to the same evaluation job was controlled from 3 to 50 in 6 steps using 10-point scale. The abilities of participants were estimated using Item Response Theory. All parameters of IRT models, which is called as Generalized Partial Credit Model, such as "ability", "consistency", and "strictness", were estimated well using MCMC technique; their standard deviation errors gradually decrease with the number of peers. As a preliminary result of optimisation, an appropriate number of peers was 15 as comparing the stadardised errors across the conditions.