Summary of the paper
| Title | A Rough Set Formalization of Quantitative Evaluation with Ambiguity |
|---|---|
| Authors | Patrick Paroubek and Xavier Tannier |
| Abstract | In this paper, we present the founding elements of a formal model of the evaluation paradigm in natural language processing. We propose an abstract model of objective quantitative evaluation based on rough sets, as well as the notion of potential performance space for describing the performance variations corresponding to the ambiguity present in hypothesis data produced by a computer program, when comparing it to the reference data created by humans. A formal model of the evaluation paradigm will be useful for comparing evaluations protocols, investigating evaluation constraint relaxation and getting a better understanding of the evaluation paradigm, provided it is general enough to be able to represent any natural language processing task. |
| Topics | Evaluation methodologies |
Full paper  |
A Rough Set Formalization of Quantitative Evaluation with Ambiguity |
| Bibtex | @InProceedings{PAROUBEK12.443, author = {Patrick Paroubek and Xavier Tannier}, title = {A Rough Set Formalization of Quantitative Evaluation with Ambiguity}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |