Summary of the paper
| Title | EVALIEX ― A Proposal for an Extended Evaluation Methodology for Information Extraction Systems |
|---|---|
| Authors | Christina Feilmayr, Birgit Pröll and Elisabeth Linsmayr |
| Abstract | Assessing the correctness of extracted data requires performance evaluation, which is accomplished by calculating quality metrics. The evaluation process must cope with the challenges posed by information extraction and natural language processing. In the previous work most of the existing methodologies have been shown that they support only traditional scoring metrics. Our research work addresses requirements, which arose during the development of three productive rule-based information extraction systems. The main contribution is twofold: First, we developed a proposal for an evaluation methodology that provides the flexibility and effectiveness needed for comprehensive performance measurement. The proposal extends state-of-the-art scoring metrics by measuring string and semantic similarities and by parameterization of metric scoring, and thus simulating with human judgment. Second, we implemented an IE evaluation tool named EVALIEX, which integrates these measurement concepts and provides an efficient user interface that supports evaluation control and the visualization of IE results. To guarantee domain independence, the tool additionally provides a Generic Mapper for XML Instances (GeMap) that maps domain-dependent XML files containing IE results to generic ones. Compared to other tools, it provides more flexible testing and better visualization of extraction results for the comparison of different (versions of) information extraction systems. |
| Topics | Evaluation methodologies, Information Extraction, Information Retrieval, Tools, systems, applications |
Full paper  |
EVALIEX ― A Proposal for an Extended Evaluation Methodology for Information Extraction Systems |
| Bibtex | @InProceedings{FEILMAYR12.204, author = {Christina Feilmayr and Birgit Pröll and Elisabeth Linsmayr}, title = {EVALIEX ― A Proposal for an Extended Evaluation Methodology for Information Extraction Systems}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |