Summary of the paper
| Title | Evaluating Dialogue Act Tagging with Naive and Expert Annotators |
|---|---|
| Authors | Jeroen Geertzen, Volha Petukhova and Harry Bunt |
| Abstract | In this paper the dialogue act annotation of naive and expert annotators, both annotating the same data, are compared in order to characterise the insights annotations made by different kind of annotators may provide for evaluating dialogue act tagsets. It is argued that the agreement among naive annotators provides insight in the clarity of the tagset, whereas agreement among expert annotators provides an indication of how reliably the tagset can be applied when errors are ruled out that are due to deficiencies in understanding the concepts of the tagset, to a lack of experience in using the annotation tool, or to little experience in annotation more generally. An indication of the differences between the two groups in terms of inter-annotator agreement and tagging accuracy on task-oriented dialogue in different domains, annotated with the DIT++ dialogue act tagset is presented, and the annotations of both groups are assessed against a gold standard. Additionally, the effect of the reduction of the tagset’s granularity on the performances of both groups is looked into. In general, it is concluded that the annotations of both groups provide complementary insights in reliability, clarity, and more fundamental conceptual issues. |
| Language | |
| Topics | Dialogue & Natural Interactivity, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper |
Evaluating Dialogue Act Tagging with Naive and Expert Annotators |
Slides |
Evaluating Dialogue Act Tagging with Naive and Expert Annotators |
| Bibtex | @InProceedings{GEERTZEN08.279, author = {Jeroen Geertzen, Volha Petukhova and Harry Bunt}, title = {Evaluating Dialogue Act Tagging with Naive and Expert Annotators}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |