Summary of the paper
| Title | Identifying Nuggets of Information in GALE Distillation Evaluation |
|---|---|
| Authors | Olga Babko-Malaya, Greg Milette, Michael Schneider and Sarah Scogin |
| Abstract | This paper describes an approach to automatic nuggetization and implemented system employed in GALE Distillation evaluation to measure the information content of text returned in response to an open-ended question. The system identifies nuggets, or atomic units of information, categorizes them according to their semantic type, and selects different types of nuggets depending on the type of the question. We further show how this approach addresses the main challenges for using automatic nuggetization for QA evaluation: the variability of relevant nuggets and their dependence on the question. Specifically, we propose a template-based approach to nuggetization, where different semantic categories of nuggets are extracted dependent on the template of a question. During evaluation, human annotators judge each snippet returned in response to a query as relevant or irrelevant, whereas automatic template-based nuggetization is further used to identify the semantic units of information that people would have selected as ‘relevant' or ‘irrelevant' nuggets for a given query. Finally, the paper presents the performance results of the nuggetization system which compare the number of automatically generated nuggets and human nuggets and show that our automatic nuggetization is consistent with human judgments. |
| Topics | Evaluation methodologies, Question Answering, Information Extraction, Information Retrieval |
Full paper  |
Identifying Nuggets of Information in GALE Distillation Evaluation |
| Bibtex | @InProceedings{BABKOMALAYA12.482, author = {Olga Babko-Malaya and Greg Milette and Michael Schneider and Sarah Scogin}, title = {Identifying Nuggets of Information in GALE Distillation Evaluation}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |