| Title |
Target Suites for Evaluating the Coverage of Text Generators |
| Authors |
Bateman John A. (Linguistics and Literature, University of Bremen, bateman@uni-bremen.de)
Hartley Anthony F. (Information Technology Research Institute, University of Brighton, UK, Tony.Hartley@itri.brighton.ac.uk) |
| Keywords |
Automatic annotation, Evaluation, Grammatical coverage, Natural Language Generation, NLG, Surface realization, Target suites, Test suites |
| Session |
Session EP1 - Evaluation and Written Area |
| Full Paper |
92.ps, 92.pdf |
| Abstract |
Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora. But they have limitations. First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation. Second, they omit phenomena such as discourse markers which are key issues in text production. We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme. |
