An empirical study of expert recommendations for the algorithm design of an intelligent study guide.

摘要

Personalized service for web-based learning has recently received considerable attention because of different needs among users. Most recommendation systems consider learner/user preferences, interests, or browsing behaviors when analyzing learner behaviors for personalized services. However, these systems neglect the importance of learner knowledge level for implementing personalized mechanisms. A major challenge for web-based educational systems is to provide students with personalized learning instructions, such as the most suitable pedagogical recommendations that best match their knowledge level.; The work presented in this thesis is a part of a larger ongoing project: the design and implementation of a web-based Adaptive Educational System, the Poly Study Guide, which can guide student knowledge remediation by making personalized assessment-driven recommendation. A study recommendation module in this study guide can be guided by an item inference engine, such as Partial Order Knowledge Structures (POKS) (Desmarais, Meshkinfam and Gagnon, 2006). Two objectives of this thesis are: to investigate the requirements for the recommendation module in the intelligent study guide, and to devise an appropriate algorithm that can grant the study guide the ability to diagnose knowledge states and make individual study recommendation.; In order to investigate the requirements for the recommendation module, we realized an experiment with eight experienced instructors to investigate the process of one-on-one tutoring. More specifically, we collected information in this experiment about professional knowledge diagnosis and study plan recommendations for each individual student, and examined the agreement among recommendations from different instructors in order to determine what desired recommendation results a study guide should deliver. Besides, we also attempted to determine the value of detailed answers for improving recommendations.; Both quantitative and qualitative approaches were employed for analysis and interpretation of the findings in this experiment. Some major findings are briefly summarized here. The agreement among recommendations from eight instructors is substantial. Thus, we can consider those results as expert recommendations and emulate them in the recommendation module of our proposed study guide. In addition, all instructors made corrections in their recommendations after they evaluated students' complete answer sheets, especially in the case of making recommendations for students in medium knowledge level. Besides, in their responses to a questionnaire, the instructors revealed strongly positive perceptions toward the value of the complete answer sheet. Thus, a computerized study guide may not make as good recommendations as a professional human instructor unless it could analyze the answers.; We devised a q-matrix and a relatively simple but effective algorithm in the recommendation module that could emulate the expert recommendations collected in the experiment. Furthermore, a simulation test was performed to validate the effectiveness of the recommendation module in our intelligent study guide. We compared the recommendations from this intelligent study guide with those from the experts.; The results of this simulation test show that the accuracy of recommendations from our program is superior to random recommendations and it increases gradually when more items are administered. When the responses to all the eight sub-question items are given, the accuracy of this recommendation module reach almost 90%. These findings can basically confirm the effectiveness of this recommendation algorithm. Besides, the results of the simulation test also confirm that more accurate the item assessment is, more accurate the recommendation is. Thus, the accuracy of recommendation is also dependent upon the effectiveness of the item assessment engine. Undoubtedly, a good item assessment engine, which can efficiently infer accurate