Robust Audio-Visual Speech Recognition System based on Gabor Features and Dynamic Stream Weight Adaption

摘要

This paper aims to enhance the performance of audio-visual speech recognition (AVSR) systems by introducing contributions in both the front-end and back-end system stages. Identifying a reliable feature is a crucial step towards enhancing the front-end stage of both audio-module and visual-module. A two-dimensional Gabor filter with different scales and directions is utilized to generate a set of noise robust spectro-temporal audio and visual features. The performance achieved from the Gabor audio features (GAFs) and Gabor visual features (GVFs) is compared to the performance of the traditional audio features such as MFCC, PLP, RASTA-PLP and visual features such as DCT2. The experimental results demonstrate that a system utilizes Gabor features in the front-end has a much better performance, especially at low SNR levels. To enhance the back-end stage, a framework based on synchronous multi-stream hidden Markov model is proposed to solve the dynamic stream weight estimation problem. To demonstrate the effect of dynamic weighting on enhancing the AVSR performance, we empirically compare between late integration (LI) and early integration (EI) strategies, especially in a low-SNR scenario. The experimental results show that the AVSR-LI system achieves superior performance for all SNR levels compared to AVSR-EI system.