Summary of the paper
| Title | Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation |
|---|---|
| Authors | Hannah Copperman and Christopher R. Walker |
| Abstract | This paper describes an interface that was developed for processing large amounts of human judgments of linguistically annotated data. Fred’s Reusable Evaluation Device (“Fred”) provides administrators with a tool to submit linguistic evaluation tasks to judges. Each evaluation task is then presented to exactly two judges, who can submit their judgments at their own leisure. Fred then provides several metrics to administrators. The most important metric is precision, which is provided for each evaluation task and each annotator. Administrators can look at precision for a given data set over time, as well as by evaluation type, data set, or annotator. Inter-annotator agreement is also reported, and that can be tracked over time as well. The interface was developed to provide a tool for evaluating semantically marked up text. The types of evaluations Fred has been used for so far include things like correctness of subject-relation identification, and correctness of temporal relations. However, Fred’s full versatility has not yet been fully exploited. |
| Topics | Tools, systems, applications, Corpus (creation, annotation, etc.), Semantics |
Full paper |
Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation |
Slides |
- |
| Bibtex | @InProceedings{COPPERMAN10.444, author = {Hannah Copperman and Christopher R. Walker}, title = {Fred’s Reusable Evaluation Device: Providing Support for Quick and Reliable Linguistic Annotation}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |