LREC 2006 - Proceedings sorted by papers

Title	An Efficient Approach to Gold-Standard Annotation: Decision Points for Complex Tasks
Authors	J. Medero, K. Maeda, S. Strassel, C. Walker
Abstract	Inter-annotator consistency is a concern for any corpus building effort relying on human annotation. Adjudication is as effective way to locate and correct discrepancies of various kinds. It can also be both difficult and time-consuming. This paper introduces Linguistic Data Consortium (LDC)’s model for decision point-based annotation and adjudication, and describes the annotation tools developed to enable this approach for the Automatic Content Extraction (ACE) Program. Using a customized user interface incorporating decision points, we improved adjudication efficiency over 2004 annotation rates, despite increased annotation task complexity. We examine the factors that lead to more efficient, less demanding adjudication. We further discuss how a decision point model might be applied to annotation tools designed for a wide range of annotation tasks. Finally, we consider issues of annotation tool customization versus development time in the context of a decision point model.
Keywords	adjudication, gold standard, decision points, tool design
Full paper	An Efficient Approach to Gold-Standard Annotation: Decision Points for Complex Tasks

Title

An Efficient Approach to Gold-Standard Annotation: Decision Points for Complex Tasks

Authors

J. Medero, K. Maeda, S. Strassel, C. Walker

Abstract

Inter-annotator consistency is a concern for any corpus building effort relying on human annotation. Adjudication is as effective way to locate and correct discrepancies of various kinds. It can also be both difficult and time-consuming. This paper introduces Linguistic Data Consortium (LDC)’s model for decision point-based annotation and adjudication, and describes the annotation tools developed to enable this approach for the Automatic Content Extraction (ACE) Program. Using a customized user interface incorporating decision points, we improved adjudication efficiency over 2004 annotation rates, despite increased annotation task complexity. We examine the factors that lead to more efficient, less demanding adjudication. We further discuss how a decision point model might be applied to annotation tools designed for a wide range of annotation tasks. Finally, we consider issues of annotation tool customization versus development time in the context of a decision point model.

Keywords

adjudication, gold standard, decision points, tool design

Full paper

An Efficient Approach to Gold-Standard Annotation: Decision Points for Complex Tasks