From expert-derived user needs to user-perceived ease of use andusefulness: A two-phase mixed-methods evaluation framework

摘要

Underspecified user needs and frequent lack of a gold standard reference are typical barriers to technology evaluation. To address this problem, this paper presents a two-phase evaluation framework involving usability experts (phase 1) and end-users (phase 2). In phase 1, a cross-system functionality alignment between expert-derived user needs and system functions was performed to inform the choice of “the best available” comparison system to enable a cognitive walkthrough in phase 1 and a comparative effectiveness evaluation in phase 2. During phase 2, five quantitative and qualitative evaluation methods are mixed to assess usability: time-motion analysis, software log, questionnaires – System Usability Scale and the Unified Theory of Acceptance of Use of Technology, think-aloud protocols, and unstructured interviews. Each method contributes data for a unique measure (e.g., time motion analysis contributes task-completion-time; software log contributes action transition frequency). The measures are triangulated to yield complementary insights regarding user-perceived ease-of-use, functionality integration, anxiety during use, and workflow impact. To illustrate its use, we applied this framework in a formative evaluation of a software called Integrated Model forPatient Care and Clinical Trials (IMPACT). We conclude that this mixed-methodsevaluation framework enables an integrated assessment of user needs satisfactionand user-perceived usefulness and usability of a novel design. This evaluationframework effectively bridges the gap between co-evolving user needs andtechnology designs during iterative prototyping and is particularly useful whenit is difficult for users to articulate their needs for technology support dueto the lack of a baseline.