Summary of the paper
| Title | Creating a Research Collection of Question Answer Sentence Pairs with Amazon’s Mechanical Turk |
|---|---|
| Authors | Michael Kaisser and John Lowe |
| Abstract | Each year NIST releases a set of question, document id, answer-triples for the factoid questions used in the TREC Question Answering track. While this resource is widely used and proved itself useful for many purposes, it also is too coarse a grain-size for a lot of other purposes. In this paper we describe how we have used Amazon’s Mechanical Turk to have multiple subjects read the documents and identify the sentences themselves which contain the answer. For most of the 1911 questions in the test sets from 2002 to 2006 and each of the documents said to contain an answer, the Question-Answer Sentence Pairs (QASP) corpus introduced in this paper contains the identified answer sentences. We believe that this corpus, which we will make available to the public, can further stimulate research in QA, especially linguistically motivated research, where matching the question to the answer sentence by either syntactic or semantic means is a central concern. |
| Language | Single language |
| Topics | Question Answering, Corpus (creation, annotation, etc.), Information Extraction, Information Retrieval |
Full paper |
Creating a Research Collection of Question Answer Sentence Pairs with Amazon’s Mechanical Turk |
Slides |
- |
| Bibtex | @InProceedings{KAISSER08.565, author = {Michael Kaisser and John Lowe}, title = {Creating a Research Collection of Question Answer Sentence Pairs with Amazon’s Mechanical Turk}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |