Summary of the paper
| Title | BLEU Evaluation of Machine-Translated English-Croatian Legislation |
|---|---|
| Authors | Sanja Seljan, Marija Brkić and Tomislav Vičić |
| Abstract | This paper presents work on the evaluation of online available machine translation (MT) service, i.e. Google Translate, for English-Croatian language pair in the domain of legislation. The total set of 200 sentences, for which three reference translations are provided, is divided into short and long sentences. Human evaluation is performed by native speakers, using the criteria of adequacy and fluency. For measuring the reliability of agreement among raters, Fleiss' kappa metric is used. Human evaluation is enriched by error analysis, in order to examine the influence of error types on fluency and adequacy, and to use it in further research. Translation errors are divided into several categories: non-translated words, word omissions, unnecessarily translated words, morphological errors, lexical errors, syntactic errors and incorrect punctuation. The automatic evaluation metric BLEU is calculated with regard to a single and multiple reference translations. System level Pearson's correlation between BLEU scores based on a single and multiple reference translations is given, as well as correlation between short and long sentences BLEU scores, and correlation between the criteria of fluency and adequacy and each error category. |
| Topics | Evaluation methodologies, Machine Translation, SpeechToSpeech Translation, Usability, user satisfaction |
Full paper  |
BLEU Evaluation of Machine-Translated English-Croatian Legislation |
| Bibtex | @InProceedings{SELJAN12.233, author = {Sanja Seljan and Marija Brkić and Tomislav Vičić}, title = {BLEU Evaluation of Machine-Translated English-Croatian Legislation}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |