A model for classification of errors and evaluation of translation quality in a Russian-English machine translation system.

摘要

The assessment of translation quality is an essential component of the overall translation process. Indeed, quality is the ultimate criterion by which any translation is measured. There is considerable difference among linguists, as well as scholars in other fields, on what constitutes a "good" translation. Moreover, there are no objective and universally applicable criteria for assessing translation quality. Inasmuch as a universally accepted, genuinely objective system has never been developed for evaluating human translations, the problem of assessing MT output is rendered considerably more acute.;This research focuses on two primary objectives: (1) a narrow one to examine various approaches to error analysis in assessing the quality of several machine translation (MT) systems, and (2) a broader one to delineate implications for improving the overall quality of both machine and human translations.;A review of the historical and theoretical literature on translation in general, and machine translation in particular, serves as a framework for the investigation of MT quality. Subsequently, several evaluative techniques that have application in assessing MT output are discussed.;Outputs from both the American SYSTRAN (Russian-to-English) and the Soviet AMPAR (English-to-Russian) MT systems are analyzed, and errors are categorized according to frequency and type, e.g., polysemanticity, lexical preference (context), case government, article, lexical error, grammatical agreement, unknown terminology, idiomaticity, word order, input error, unknown abbreviation, pronominal reference, anaphora, and noun groups. The appendix includes a comprehensive corpus of raw and edited output from the SYSTRAN system.;A proposed model is presented for evaluating MT output based on the primary characteristics of accuracy, intelligibility, and style. Thus, errors are classified according to type and frequency of occurrence as a means of identifying quantifiable elements for inclusion in an evaluation grid. For example, the model assesses a translation on the basis of each of these criteria and assigns a rating at one of five possible levels: Fully Professional (L-5), Advanced Professional (L-4), Intermediate Professional (L-3), Limited Professional (L-2), and Elementary (L-1).;The findings strongly indicate that the proper resolution of context--perhaps through artificial intelligence techniques--appears to be the sine qua non for MT systems in the continuing pursuit of fully automatic high quality translation.