Multimodal System for Diagnosis and Polysensory Stimulation of Subjects with Communication Disorders

摘要

An experimental multimodal system, designed for polysensory diagnosis and stimulation of persons with impaired communication skills or even non-communicative subjects is presented. The user interface includes an eye tracking device and the EEG monitoring of the subject. Furthermore, the system consists of a device for objective hearing testing and an autostereoscopic projection system designed to stimulate subjects through their immersion in a virtual environment. Data analysis methods are described, and experiments associated with classification of mental states during listening exercises as well as audio-visual stimuli are presented and discussed. Feature extraction was based on discrete wavelet transformation and clustering employing the k-means algorithm was designed. All algorithms were implemented in the Python programming language with the use of Open Source libraries. Tests of the proposed system were performed in a Special School and Educational Center in Kos-cierzyna, Poland. Results and comparison with data gathered from the control group of healthy people are presented and discussed.