Multimodal interaction means computer operators can communicate naturallyand intuitively with the system by using modalities such as speech and gesture,facilitating complex spatial tasks, such as air-traffic control. Measuring theircognitive load in real-time allows the system to adapt to users affected by highcognitive load, easing the demand and avoiding stress, frustration and errors. Thisdissertation explores the viability of using features extracted from multimodalinteractive data as symptomatic cues of high cognitive load.Two empirical user studies were conducted to collect multimodal interactivedata under levels of increasing load, in a traffic management scenario. A novelframework to collect natural, unbiased multimodal input is presented, addressingthe requirements for designing multimodal tasks of varying complexity.The first study uses a speech and manual gesture interface, and examineschanges in conceptual communicative structures, namely the pattern of semanticredundancy and complementarity. The results confirm that people are moresemantically redundant when load is low; and more semantically complementaryduring high load tasks. Consistent with modal models of working memory, peoplemanage high levels of load by diffusing communication across different modalities,with the least duplication possible to effectively expand their available workingmemory resources.The second, longitudinal study used a pen-gesture and speech interface, andexamined changes to communication structures at the production level, correlatingthe degree of modal degradation to cognitive load. The results show thatmodal input degrades to a greater degree during high load tasks than during lowload tasks. The use of cognitive tools also increases as load increases, revealingyet another type of index.The feasibility of using multimodal interaction features as indices of cognitiveload is validated, future work should be geared toward assessing their sensitivityand diagnostic value.
展开▼
