Cognitive Augmented and Mixed Reality: A New Era of Interaction

摘要

This talk will examine the implications of cognitive Augmented and Mixed Reality (AR/MR) technologies on information systems engineering and their effect on personalized, immersive computing. AR/MR technologies facilitate interactions with users involving enhanced 3D vision, spatial hearing, hands and arms gestures, and vocal interactions, as well as controlling information flow of IoT devices monitoring and controlling the physical surrounding environment. Given the complexity of these personalized immersive interactions, a change is needed in information systems engineering, particularly in requirements engineering and quality assurance practices. Requirements gathering and validation of users' experience implies that development engineers need to experience, understand, and use these new AR/MR modes of interactions. One wonders how would engineers perform activities such as shadowing, recording and analyzing users' actions, fully understand gestures and gazing intent, factoring eyesight limitations, depth perceptions, and more. Virtual Reality (VR) uses comprehensive digital environment devices and digital cues. However, AR/MR require hands-free for interacting with real world environment in which there are no pointers nor cues. Repeated VR digital world can track and monitor user's actions and derive usability statistics. Real environments behave according to the laws of nature and users' personalized parameters in addition to the interacted digital information overlay. AR and MR are meant for augmenting users' eyesight, audio, gestures and cognitive collaboration with the IT systems, whilst developers are required to create generic software solutions that abstracts these cognitive interactions as well as encapsulate hardware complexity such as the camera field-of-view versus the person, or wearables' comfort levels. How should requirements engineering and quality assurance disciplines evolve to assist developers in understanding how and what needs to be improved, added, or removed in AR/MR driven systems? How can we perform agile enhancements in UX and UI, normalize, relate and transform the input provided from field experience, to practical actions? How should the development environment look like when dealing with AR/MR systems? What will be the new form of unit testing? There is a need for research to answer these questions. Examples will be given from different domains such as industrial support and maintenance, automotive and aerospace manufacturing, and more.