Summary of the paper
| Title | Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies |
|---|---|
| Authors | Jana Z. Sukkarieh and Eleanor Bolge |
| Abstract | Automatic content scoring for free-text responses has started to emerge as an application of Natural Language Processing in its own right, much like question answering or machine translation. The task, in general, is reduced to comparing a student’s answer to a model answer. Although a considerable amount of work has been done, common benchmarks and evaluation measures for this application do not currently exist. It is yet impossible to perform a comparative evaluation or progress tracking of this application across systems ― an application that we view as a textual entailment task. This paper concentrates on introducing an Educational Testing Service-built test suite that makes a step towards establishing such a benchmark. The suite can be used as regression and performance evaluations both intra-c-rater® or inter automatic content scoring technologies. It is important to note that existing textual entailment test suites like PASCAL RTE or FraCas, though beneficial, are not suitable for our purposes since we deal with atypical naturally-occurring student responses that need to be categorized in order to serve as regression test cases. |
| Topics | Textual Entailment and Paraphrasing, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper |
Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies |
Slides |
- |
| Bibtex | @InProceedings{SUKKARIEH10.310, author = {Jana Z. Sukkarieh and Eleanor Bolge}, title = {Building a Textual Entailment Suite for the Evaluation of Automatic Content Scoring Technologies}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |