Kinect-Sign: Teaching Sign Language to 'Listeners' through a Game

摘要

Sign language is the hearing impaired form of communicating with other people, including listeners. Most cases, impaired people have learned sign language form childhood. The problem arises when a listener comes in contact with an impaired person. For instances, if a couple has a child which is impaired, the parents find a challenge to learn the sign language. In this article, a new playful approach to assist the listeners to learn sign language is proposed. This proposal is a serious game composed of two modes: School-mode and Competition-mode. The first offers a virtual school where the user learns to sign letters and the second offers an environment towards applying the learned letters. Behind the scenes, the proposal contains a sign language recognition system, based on three modules: 1 - the standardization of the Kinect depth camera data; 2 - a gesture library relying on the standardized data; and 3 - the real-time recognition of gestures. A prototype was developed - Kinect-Sign - and tested in a Portuguese Sign-Language school and on eNTERFACE'13 resulting in a joyful acceptance of the approach.