| LREC 2000 2nd International Conference on Language Resources & Evaluation | ||||||
|
|
|
|
|
|
|
| Title | Coreference Resolution Evaluation Based on Descriptive Specificity |
| Authors | Trouilleux François (Xerox Research Centre Europe, 6 Chemin de Maupertuis. 38240 Meylan) Gaussier Eric (Xerox Research Centre Europe, 6 Chemin de Maupertuis. 38240 Meylan) Bès Gabriel G. (Groupe de recherche dans les industries de la langue (GRIL), UFR-LACC, Université Blaise-Pascal, Clermont 2, 34, avenue Carnot. 63037 Clermont-Ferrand - France) Zaenen Annie (Xerox Research Centre Europe, 6 Chemin de Maupertuis. 38240 Meylan) |
| Keywords | Anaphora, Coreference, Descriptive Specificity, Evaluation, Text Understanding |
| Session | Session EO4 - Grammars and Systems Evaluation |
| Full Paper | 131.ps, 131.pdf |
| Abstract | This paper introduces a new evaluation method for the coreference resolution task. Considering that coreference resolution is a matter of linking expressions to discourse referents, we set our evaluation criteron in terms of an evaluation of the denotations assigned to the expressions. This criterion requires that the coreference chains identified in one annotation stand in a one-to-one correspondence with the coreference chains in the other. To determine this correspondence and with a view to keep closer to what human interpretation of the coreference chains would be, we take into account the fact that, in a coreference chain, some expressions are more specific to their referent than others. With this observation in mind, we measure the similarity between the chains in one annotation and the chains in the other, and then compute the optimal similarity between the two annotations. Evaluation then consists in checking whether the denotations assigned to the expressions are correct or not. New measures to analyse errors are also introduced. A comparison with other methods is given at the end of the paper. |