A Study Examining a Real-Time Sign Language-to-Text Interpretation System Using Crowdsourcing

摘要

The current study examined how to use crowdsourcing to convert sign language-to-text. Generally in Japan, a sign language interpreter reads and vocalizes the sign language of the speaker, and caption typists generate captions from the vocalization. However, this method doubles labor costs and delays caption provision. Therefore, we developed a system that interprets sign language-to-caption text via crowdsourcing, with non-experts performing interpretations. While many individuals classified as dcaf/hard-of-hearing (DHH) who can read sign language are suitable for this task, not all of them possess adequate typing skills. To address this, our system divides live sign language video into shorter segments, distributing them to workers. After the worker interprets and types the segments to text, the system generates captions through integration of these texts. Furthermore, we provide a user interface for playback speed control and one second rewinding in order to improve the ease with which tasks are completed. Our system can establish an environment that not only allows the interpretation of sign language-to-caption text, but also provides an opportunity for DHH individuals to assist those that are unable read sign language. We conducted a test using our prototype system for sign language-to-text interpretation. The mean time it took a worker to finish a task was 26 s for a 9 s segment. The combined total rate of missing text and collision between segments was 66%. Analysis of questionnaire responses found that workers assigned fewer tasks considered the tasks more enjoyable.