An adaptive integration method based on product HMM for Bi-modal speech recognition

摘要

In recent years, there has been higher demands for Antomatic Speech recognition system operated robustly in the various noisy environments. Therefore, many researchers have interest in the bimodal speech recognition by using not only the audio but also the visual information extracted from the sequence of the speaker's lip images. To realize the bimodal speech recognition, it is important to integrate effectively the audio and visual information. In integrating them, "(1) Synchronization of the audio and visual information, (2) Adaptability of the system, adjusting to changes in environment" are important issues. In the problem of (1), each feature of the speech and lip movement has the time lag, and has the correlation. For such the problem, we introduce the integration method using HMM composition. In (2), we have examined that the stream weight can be adaptively estimated by GPD algorithm. The evaluation experiment shows that the proposed method improves recognition accuracy of noisy speech.