Summary of the paper
| Title | Evaluating Robustness Of A QA System Through A Corpus Of Real-Life Questions |
|---|---|
| Authors | Laurianne Sitbon, Patrice Bellot and Philippe Blache |
| Abstract | This paper presents the sequential evaluation of the question answering system SQuaLIA. This system is based on the same sequential process as most statistical question answering systems, involving 4 main steps from question analysis to answer extraction.The evaluation is based on a corpus made from 20 questions taken in the set of an evaluation campaign and which were well answered by SQuaLIA. Each of the 20 questions has been typed by 17 native participants, non natives and dyslexics. They were vocally instructed the target of each question. Each of the 4 analysis steps of the system involves a loss of accuracy, until an average of 60 of right answers at the end of the process. The main cause of this loss seems to be the orthographic mistakes users make on nouns. |
| Language | Single language |
| Topics | Corpus (creation, annotation, etc.), Question Answering, Usability, user satisfaction |
Full paper |
Evaluating Robustness Of A QA System Through A Corpus Of Real-Life Questions |
Slides |
- |
| Bibtex | @InProceedings{SITBON08.244, author = {Laurianne Sitbon, Patrice Bellot and Philippe Blache}, title = {Evaluating Robustness Of A QA System Through A Corpus Of Real-Life Questions}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |