LREC 2000 - Papers

LREC 2000 2^nd International Conference on Language Resources & Evaluation

Conference Papers

Papers by paper title: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Papers by ID number: 1-50, 51-100, 101-150, 151-200, 201-250, 251-300, 301-350, 351-377.

List of all papers and abstracts.

Previous Paper Next Paper

Title On the Usage of Kappa to Evaluate Agreement on Coding Tasks

Authors Di Eugenio Barbara (Electrical Engineering and Computer Science, University of Illinois at Chicago, Chicago, IL 60607, USA, dieugeni@eecs.uic.edu)

Keywords Annotated Corpora, Intercoder Reliability

Session Session SP2 - Spoken Language Resources Issues from Construction to Validation

Abstract In recent years, the Kappa coefficient of agreement has become the de facto standard to evaluate intercoder agreement in the discourse and dialogue processing community. Together with the adoption of this standard, researchers have adopted one specific scale to evaluate Kappa values, the one proposed in (Krippendorff, 1980). In this paper, I highlight some issues that should be taken into account when evaluating Kappa values. Finally, I speculate on whether Kappa could be used as a measure to evaluate a system’s performance.

Title	On the Usage of Kappa to Evaluate Agreement on Coding Tasks
Authors	Di Eugenio Barbara (Electrical Engineering and Computer Science, University of Illinois at Chicago, Chicago, IL 60607, USA, dieugeni@eecs.uic.edu)
Keywords	Annotated Corpora, Intercoder Reliability
Session	Session SP2 - Spoken Language Resources Issues from Construction to Validation
Abstract	In recent years, the Kappa coefficient of agreement has become the de facto standard to evaluate intercoder agreement in the discourse and dialogue processing community. Together with the adoption of this standard, researchers have adopted one specific scale to evaluate Kappa values, the one proposed in (Krippendorff, 1980). In this paper, I highlight some issues that should be taken into account when evaluating Kappa values. Finally, I speculate on whether Kappa could be used as a measure to evaluate a system’s performance.