Summary of the paper
| Title | Analyzing the Impact of Prevalence on the Evaluation of a Manual Annotation Campaign |
|---|---|
| Authors | Karën Fort, Claire François, Olivier Galibert and Maha Ghribi |
| Abstract | This article details work aiming at evaluating the quality of the manual annotation of gene renaming couples in scientific abstracts, which generates sparse annotations. To evaluate these annotations, we compare the results obtained using the commonly advocated inter-annotator agreement coefficients such as S, κ and π, the less known R, the weighted coefficients κω and α as well as the F-measure and the SER. We analyze to which extent they are relevant for our data. We then study the bias introduced by prevalence by changing the way the contingency table is built. We finally propose an original way to synthesize the results by computing distances between categories, based on the produced annotations. |
| Topics | Evaluation methodologies, Corpus (creation, annotation, etc.), Validation of LRs |
Full paper  |
Analyzing the Impact of Prevalence on the Evaluation of a Manual Annotation Campaign |
| Bibtex | @InProceedings{FORT12.549, author = {Karën Fort and Claire François and Olivier Galibert and Maha Ghribi}, title = {Analyzing the Impact of Prevalence on the Evaluation of a Manual Annotation Campaign}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |