Arabic sign language recognition using vision and hand tracking features with HMM

摘要

Sign language employs signs made by hands and facial expressions to convey meaning. Sign language recognition facilitates the communication between community and hearing-impaired people. This work proposes a recognition system for Arabic sign language using four types of features, namely modified Fourier transform, local binary pattern, histogram of oriented gradients, and a combination of histogram of oriented gradients and histogram of optical flow. These features are evaluated using hidden Markov model on two databases. The best performance is achieved with modified Fourier transform and histogram of oriented gradients features with 99.11% and 99.33% accuracies, respectively. In addition, two algorithms are proposed, one for segmenting sign video streams captured by Microsoft Kinect V2 into signs and the second for hand detection in video streams. The obtained results show that our algorithms are efficient in segmenting sign video streams and detecting hands in video streams.