Summary of the paper
| Title | Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk |
|---|---|
| Authors | Fang Xu and Dietrich Klakow |
| Abstract | Creating more fine-grained annotated data than previously relevent document sets is important for evaluating individual components in automatic question answering systems. In this paper, we describe using the Amazon's Mechanical Turk (AMT) to judge whether paragraphs in relevant documents answer corresponding list questions in TREC QA track 2004. Based on AMT results, we build a collection of 1300 gold-standard supporting paragraphs for list questions. Our online experiments suggested that recruiting more people per task assures better annotation quality. In order to learning true labels from AMT annotations, we investigated three approaches on two datasets with different levels of annotation errors. Experimental studies show that the Naive Bayesian model and EM-based GLAD model can generate results highly agreeing with gold-standard annotations, and dominate significantly over the majority voting method for true label learning. We also suggested setting higher HIT approval rate to assure better online annotation quality, which leads to better performance of learning methods. |
| Topics | Corpus (creation, annotation, etc.), Question Answering, Statistical and machine learning methods |
Full paper |
Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk |
Slides |
- |
| Bibtex | @InProceedings{XU10.241, author = {Fang Xu and Dietrich Klakow}, title = {Paragraph Acquisition and Selection for List Question Using Amazon’s Mechanical Turk}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |