Machine and Social Intelligent Peer-Assessment Systems for Assessing Large Student Populations in Massive Open Online Education

摘要

The European étoile project aims to create high quality free open education in complex systems science, including quality assured certification. Universities and colleges worldwide increasingly use online platforms to offer courses open to the public. Massive Open Online Courses (MOOCs) give millions of people access to education from prestigious universities. Although some courses provide certification of attendance and completion, most do not provide any academic or professional recognition since this would imply a rigorous and complete evaluation of the student's achievements. Since the number of students enrolled may exceed tens of thousands, it is impractical for a lecturer (or group of lecturers) to evaluate all students using conventional hand marking. To be scalable, assessment must be automated. State-of-the-art automated assessment includes multiple choice questions and intelligent marking techniques (involving complex semantic analysis). However, none of these alone can cope with very large student populations of students and guarantee the evaluation quality required for higher education. The goal of this research is to create and evaluate a computer mediated social interaction system for massive online learning communities. This must be scalable and able to assess fairly and accurately student coursework and examinations. We call this approach "machine and socially intelligent peer assessment". We describe our system and illustrate its application. It combines peer assessment and reputation systems to provide independent computerised assessment. Assignment of student markers to scripts is based on reputation scores which emerge from their marking behaviour. A simulation experiment shows how reputation-based social structure evolves in our peer marking.system. A pilot experiment with ninety 16-year old high school students in Colombia tested the marking accuracy of our system by comparing the statistical differences between teacher-marked 'gold standard' scores, peer assessment using average scores, and our intelligent reputation-based peer assessment. The research question is to what extent does the proposed approach improve peer marking in terms of marking accuracy and fairness? We report the first results of this experiment, summarise the lessons learned, and describe further work.