Summary of the paper
| Title | Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank |
|---|---|
| Authors | Marie Mikulová and Jan Štěpánek |
| Abstract | In this paper, we present several ways to measure and evaluate the annotation and annotators, proposed and used during the building of the Czech part of the Prague Czech-English Dependency Treebank. At first, the basic principles of the treebank annotation project are introduced (division to three layers: morphological, analytical and tectogrammatical). The main part of the paper describes in detail one of the important phases of the annotation process: three ways of evaluation of the annotators - inter-annotator agreement, error rate and performance. The measuring of the inter-annotator agreement is complicated by the fact that the data contain added and deleted nodes, making the alignment between annotations non-trivial. The error rate is measured by a set of automatic checking procedures that guard the validity of some invariants in the data. The performance of the annotators is measured by a booking web application. All three measures are later compared and related to each other. |
| Topics | Corpus (creation, annotation, etc.), Evaluation methodologies, LR national/international projects, organizational/policy issues |
Full paper |
Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank |
Slides |
- |
| Bibtex | @InProceedings{MIKULOV10.388, author = {Marie Mikulová and Jan Štěpánek}, title = {Ways of Evaluation of the Annotators in Building the Prague Czech-English Dependency Treebank}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |