Pedagogical dilemmas in dynamic assessment situations: perspectives on video data from simulator-based competence tests

摘要

This study investigates navigation instructors' explanations of dynamic assessment practices during simulator-based competency tests, adopting a video-stimulated recall method. Episodes of authentic video materials from simulator-based competency tests are selected and subjected to interaction analysis. In the next step, the episodes are used for conducting stimulated recall interviews with navigation instructors (n = 11) in two focus groups. The results reveal the dynamic nature of assessing competence as well as how instructors participating in focus groups identified and critically discussed a variety of pedagogical dilemmas related to these dynamics. These are related to aspects that relate to what constitutes a valid exam question, how to assess students' responses, and consistency and fairness of competence tests. In particular, the results indicate the complexity of conducting valid and reliable assessments of knowledge-in-action in situ as well as how thoughtful scenario designs could reduce inconsistencies and unequal treatment of students. The results also highlight how a repeated and collaborative viewing of videos was helpful for instructors to identify potential problems in the dynamic assessment situations they viewed. The implications of the results highlight the need for conducting high-stake assessments for maritime certificates based only on observable behavior and video records of competence tests rather than during ongoing simulator tests. Lastly, the need for continuous and structured pedagogical development of instructors is identified in order to support their complex work of training and assessing competence.