Equivalent representations of multimodal user interfaces: Runtime Reification of Abstract User Interface Descriptions

摘要

While providing non-visual access to graphical user interfaces has been a topic of research for over 20 years, blind users still face many obstacles when using computer systems. Furthermore, daily life has become more and more infused with devices that feature some kind of visual interface. Existing solutions for providing multimodal user interfaces that ensure accessibility are largely based on either graphical toolkit hooks, queries to the application and environment, scripting, model-driven user interface development or runtime adaptation. Parallel User Interface Rendering (PUIR) is a novel approach based on past and current research into accessibility, promoting the use of abstract user interface descriptions. Based on a single consistent conceptual model, PUIR provides a mechanism to render a user interface simultaneously in multiple modalities. Each representation of the UI provides equivalent semantics to ensure that collaboration between users of different modalities is facilitated. The possible application of this novel technique goes well beyond the realm of accessibility, as multimodal interfaces become more popular and even needed. The design presented here has been implemented as a prototype for testing and further research in this fascinating area of HCI.