MEASURING HUMAN READABILITY OF MACHINE GENERATED TEXT: THREE CASE STUDIES IN SPEECH RECOGNITION AND MACHINE TRANSLATION

摘要

We present highlights from three experiments that test the readability of current state-of-the art system output from (1) an automated English speech-to-text system (2) a text-based Arabic-to-English machine translation system and (3) an audio-based Arabic-to-English MT process. We measure readability in terms of reaction time and passage comprehension in each case, applying standard psycholinguistic testing procedures and a modified version of the standard Defense Language Proficiency Test for Arabic called the DLPT. We learned that: (1) subjects are slowed down about 25percent when reading system STT output, (2) text-based MT systems enable an English speaker to pass Arabic Level 2 on the DLPT and (3) audio-based MT systems do not enable English speakers to pass Arabic Level 2. We intend for these generic measures of readability to predict performance of more application-specific tasks.