A combination of static and stroke gesture with speech for multimodal interaction in a virtual environment

摘要

The computers nowadays are powerful enough to make the virtual reality technology more practical. Virtual reality is expected to have a bright future due to the rise of several virtual reality head mounted device such as low cost Google Cardboard, Samsung Gear VR, Oculus, HTC Vive and many others. Since virtual reality had been growing fast in this few years thus it is necessary to explore new approaches for user to interact with the virtual reality more naturally. It is an important research in HCI to make the interactions with computers to be as natural as the interaction between humans. One of the approaches is to allow user to interact with the virtual reality through gesture and speech input. This paper presents a virtual reality environment with multimodal interaction technique. It allows user to interact with the virtual reality system with the static and stroke hand gesture along with speech. This multimodal interaction technique is able to perform few functions such as select, move, scale, rotate, copy, mirror, delete, check the shape, check and change the colour of an object.