Designing Non-Visual Multimodal Dialogues to SupportInformation Access for the Visually Impaired

摘要

A multimodal system accepting touchpad gesture input and speech input has been developed to support browsing onlinernnewspapers for the visually impaired. The issue addressed in this work is how to combine the two input modalities for nonvisualrntext-information browsing. An exploratory study with fourteen sighted subjects suggests that: (1) A dependencernbetween users’ choice of input modality and the type of input operations undertaken exists – speech is preferred for abstractrninput and gesture for location-based navigations. (2) Continuous repetitive operations are more likely to be performed usingrnthe touchpad. (3) Among several factors including changes in operation types, input failures, and interruptions in taskrnperformance, the first factor is the major cause of users’ input mode mixes. (4) Prevalent user strategies for error correctionsrndo not involve input modality switches but repetitions of failed operations and tryouts of different operations in the samerninput modality. (5) Training orders have limited effect on users’ preference on input modalities in later usage. (6) Skipping isrncommon in a familiar information organization. In speech, the main skipping strategy is speaking the desired item directly.rnOn the touchpad the main strategies include approximating the location of a desired item and sequential skips. Arnconfirmative experiment will be conducted with visually impaired users and will lead to suggestions for designing non-visualrnmultimodal input for the addressed application domain.