Summary of the paper
| Title | Iterative Refinement and Quality Checking of Annotation Guidelines ― How to Deal Effectively with Semantically Sloppy Named Entity Types, such as Pathological Phenomena |
|---|---|
| Authors | Udo Hahn, Elena Beisswanger, Ekaterina Buyko, Erik Faessler, Jenny Traumüller, Susann Schröder and Kerstin Hornbostel |
| Abstract | We here discuss a methodology for dealing with the annotation of semantically hard to delineate, i.e., sloppy, named entity types. To illustrate sloppiness of entities, we treat an example from the medical domain, namely pathological phenomena. Based on our experience with iterative guideline refinement we propose to carefully characterize the thematic scope of the annotation by positive and negative coding lists and allow for alternative, short vs. long mention span annotations. Short spans account for canonical entity mentions (e.g., standardized disease names), while long spans cover descriptive text snippets which contain entity-specific elaborations (e.g., anatomical locations, observational details, etc.). Using this stratified approach, evidence for increasing annotation performance is provided by kappa-based inter-annotator agreement measurements over several, iterative annotation rounds using continuously refined guidelines. The latter reflects the increasing understanding of the sloppy entity class both from the perspective of guideline writers and users (annotators). Given our data, we have gathered evidence that we can deal with sloppiness in a controlled manner and expect inter-annotator agreement values around 80% for PathoJen, the pathological phenomena corpus currently under development in our lab. |
| Topics | Corpus (creation, annotation, etc.), Named Entity recognition, Metadata |
Full paper  |
Iterative Refinement and Quality Checking of Annotation Guidelines ― How to Deal Effectively with Semantically Sloppy Named Entity Types, such as Pathological Phenomena |
| Bibtex | @InProceedings{HAHN12.755, author = {Udo Hahn and Elena Beisswanger and Ekaterina Buyko and Erik Faessler and Jenny Traumüller and Susann Schröder and Kerstin Hornbostel}, title = {Iterative Refinement and Quality Checking of Annotation Guidelines ― How to Deal Effectively with Semantically Sloppy Named Entity Types, such as Pathological Phenomena}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |