A study of methods of evaluating multimedia materials for language learning

摘要

There is a long tradition of paper based materials evaluation in ELT, but at this juncture, a scarcity of studies on ELT Multimedia (MM) materials evaluation. Such studies as have been undertaken have tended to adopt the perspective of the materials developer rather than the end user. But there have been no developed studies of evaluation methods which could be adopted by potential users. Despite calls being made for systematic evaluation, not many proposals have been developed, and there has been little exploration of potential best practice, or of the 'goodness of fit' between methods and evaluation puposes. This study aims to investigate evaluation methods in order to establish best practice in the evaluation of multimedia CALL applications, with a focus on learners' experience and opinions, and with the aim of enabling potential teacher-users of CALL materials to gauge the suitability of materials for their learners. Chapter one of this thesis provides a rationale for this study and an overview of the background to this research. Chapter two presents a review of literature undertaken in four domains: educational evaluation and research methods; Human Computer Interaction (HCI) usability evaluation methods; ELT materials evaluation; and studies ofCALL materials in use. Chapter three focuses on the design and conduct of the study by explaining how some methods of evaluation were trialled in a pilot study and four were selected for adoption and analysis in the main study. The methods selected were Foeu.r Groups, Rttrospective Protorolr, PLUM and SUM! Questionnaires and Activity Monitonitg. To determine the qualities and limitations of these methods, a set of criteria was developed from the literature on software usability evaluation methods in HCI and a broader literature on educational evaluation and research methods. The four data study chapters (4-7) each discuss one of the chosen methods and descnbe how the method was ,6perationalised in an evaluation of learner responses to multimedia software. The final chapter draws together the discussion of the findings and presents different proposals for best practice. The focus in the discussion of findings is on how the chosen methods performed according to the set of criteria. The findings confirm that focus groups and questionnaires are quick and efficient methods whereas retrospective protocols and activity monitoring provide more detailed and protracted data. Teacher evaluators can be guided by the objectives of their evaluation to explore different combinations ofthese methods. Participants in the pilot were 12 ESL students from the University of Warwick and in the main study 45 Freshman/sophomore students from a university in Pakistan. The materials used to operationalise methods were the EASE CD-ROMS listening to Lectures and Seminar Skills 1: Presentations. The research contributes to the field in undertaking an in-depth and extensive study of evaluation methods applicable to CALL materials, whi~ adopt a leamer-centred perspective, and conform to sound principles within educational evaluation, yet which draw on practice in the field of HCI, since this expertise is so relevant in the rapid development of multimedia materials for use in ELT. Moreover, by developing the composite set of core criteria this study has created a tool which practitioners in the field can use to select most appropriate methods for their particular evaluation purposes.