Multimodal approach to image perception of histology for the blind or visually impaired

摘要

Currently there is no suitable substitute technology to enable blind or visually impaired people (BVI) to interpret visual scientific data commonly generated during lab experimentation in real time, such as performing light microscopy, spectrometry, and observing chemical reactions. This reliance upon visual interpretation of scientific data certainly impedes BVIs from advancing in careers in medicine, biology and chemistry. To address this challenge, a real-time multimodal image perception system is developed to transform the standard lab blood smear image for persons with BVI to perceive, employing a combination of auditory, haptic, and vibrotactile feedbacks. These sensory feedbacks are used to convey visual information in appropriate perceptual channels, thus creating a palette of multimodal, sensorial information. A Bayesian network is developed to characterize images through two groups of features of interest: primary and peripheral features. Then, a method is conceived for optimal matching between primary features and sensory modalities. Experimental results confirmed this real-time approach of higher accuracy in recognizing and analyzing objects within images compared to tactile papers.