LREC 2000 2nd International Conference on Language Resources & Evaluation
 

Previous Paper   Next Paper

Title Target Suites for Evaluating the Coverage of Text Generators
Authors Bateman John A. (Linguistics and Literature, University of Bremen, bateman@uni-bremen.de)
Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK, Tony.Hartley@itri.brighton.ac.uk)
Keywords Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites
Session Session EP1 - Evaluation and Written Area
Full Paper 92.ps, 92.pdf
Abstract Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.