Haptic Conversion Using Detected Sound Event in Home Monitoring System for the Hard-of-Hearing

摘要

In this paper, we present a haptic conversion method using detected sound event in home monitoring assistant system for the deaf and hard-of-hearing. In the home monitoring assistant system, sounds generated in the home environment are automatically detected by sound event detection based on wireless acoustic sensor networks. And the detected sound event is converted into text and haptic vibration, and provided to help people with hearing loss. The proposed approach is mainly composed of four modules, including signal estimation based on packet loss concealment, reliable sensor channel selection using a multi-channel cross-correlation coefficient, sound event detection using bidirectional gated recurrent neural networks, and sound-to-haptic effect conversion based on percussive-harmonic source separation. Experiments show that the hard-of-hearing users were receptive to the effect of the proposed haptic conversion using the sound event detected by bidirectional gated recurrent neural networks in the home monitoring system.