Integrating Dynamic Information with Multi-stream HMM in Ultrasound-based Silent Speech Interface

摘要

In continuous-speech phone recognition, dynamic information which represents the continuity of speech plays a significant role. However, compared with feature extraction and fusion strategy, the integration of dynamic information has been paid fewer attention. In this paper, we looked into a novel method to integrate dynamic information in an ultrasound-based silent speech interface. In our method, to weight the reliabilities of the static and dynamic visual feature information respectively, we have adopted the Multi-stream Hidden Markov Model (MSHMM) technique. We evaluated our multi-stream-based integration method on a mandarin multimodal database, and compared with the traditional concatenation fusion method, our approach brings an improvement to the phonetic decoding accuracy.