Summary of the paper
| Title | The Quaero Evaluation Initiative on Term Extraction |
|---|---|
| Authors | Thibault Mondary, Adeline Nazarenko, Haïfa Zargayouna and Sabine Barreaux |
| Abstract | The Quaero program has organized a set of evaluations for terminology extraction systems in 2010 and 2011. Three objectives were targeted in this initiative: the first one was to evaluate the behavior and scalability of term extractors regarding the size of corpora, the second goal was to assess progress between different versions of the same systems, the last one was to measure the influence of corpus type. The protocol used during this initiative was a comparative analysis of 32 runs against a gold standard. Scores were computed using metrics that take into account gradual relevance. Systems produced by Quaero partners and publicly available systems were evaluated on pharmacology corpora composed of European Patents or abstracts of scientific articles, all in English. The gold standard was an unstructured version of the pharmacology thesaurus used by INIST-CNRS for indexing purposes. Most systems scaled with large corpora, contrasted differences were observed between different versions of the same systems and with better results on scientific articles than on patents. During the ongoing adjudication phase domain experts are enriching the thesaurus with terms found by several systems. |
| Topics | Evaluation methodologies, Corpus (creation, annotation, etc.) |
Full paper  |
The Quaero Evaluation Initiative on Term Extraction |
| Bibtex | @InProceedings{MONDARY12.812, author = {Thibault Mondary and Adeline Nazarenko and Haïfa Zargayouna and Sabine Barreaux}, title = {The Quaero Evaluation Initiative on Term Extraction}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |