Summary of the paper
| Title | Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations |
|---|---|
| Authors | Christian Federmann |
| Abstract | We describe a focused effort to investigate the performance of phrase-based, human evaluation of machine translation output achieving a high annotator agreement. We define phrase-based evaluation and describe the implementation of Appraise, a toolkit that supports the manual evaluation of machine translation results. Phrase ranking can be done using either a fine-grained six-way scoring scheme that allows to differentiate between ""much better"" and ""slightly better"", or a reduced subset of ranking choices. Afterwards we discuss kappa values for both scoring models from several experiments conducted with human annotators. Our results show that phrase-based evaluation can be used for fast evaluation obtaining significant agreement among annotators. The granularity of ranking choices should, however, not be too fine-grained as this seems to confuse annotators and thus reduces the overall agreement. The work reported in this paper confirms previous work in the field and illustrates that the usage of human evaluation in machine translation should be reconsidered. The Appraise toolkit is available as open-source and can be downloaded from the author's website. |
| Topics | Tools, systems, applications, Machine Translation, SpeechToSpeech Translation, Evaluation methodologies |
Full paper |
Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations |
Slides |
- |
| Bibtex | @InProceedings{FEDERMANN10.197, author = {Christian Federmann}, title = {Appraise: An Open-Source Toolkit for Manual Phrase-Based Evaluation of Translations}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |