An Architecture for Massive Essays Evaluations

摘要

The praxis of submitting an essay to an online site to be evaluated is increasing as an alternative to hand it over in a sheet of paper to the teacher. In this paper, we describe an architecture for a Massive Open Online Essays Evaluations (MO2E2) which allows us to concurrently monitor several graders and their performance. Furthermore, we carry out some statistical tests on their evaluations to compare them. Therefore, our model quantifies the reliability and the concordance among graders. In the obtained results we highlight the disagreement between the evaluators, specifically for Competence 1, with a low degree of correlation, and unsatisfactory performance in the Quadratic Weighted Kappa (QWK) analysis. Competencies that often denote a sharp discrepancy are a sign of the need for retraining to align the evaluators.