Summary of the paper
| Title | Fine-grained Linguistic Evaluation of Question Answering Systems |
|---|---|
| Authors | Sarra El Ayari, Brigitte Grau and Anne-Laure Ligozat |
| Abstract | Question answering systems are complex systems using natural language processing. Some evaluation campaigns are organized to evaluate such systems in order to propose a classification of systems based on final results (number of correct answers). Nevertheless, teams need to evaluate more precisely the results obtained by their systems if they want to do a diagnostic evaluation. There are no tools or methods to do these evaluations systematically. We present REVISE, a tool for glass box evaluation based on diagnostic of question answering system results. |
| Topics | Question Answering, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper |
Fine-grained Linguistic Evaluation of Question Answering Systems |
Slides |
- |
| Bibtex | @InProceedings{ELAYARI10.520, author = {Sarra El Ayari and Brigitte Grau and Anne-Laure Ligozat}, title = {Fine-grained Linguistic Evaluation of Question Answering Systems}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |