Summary of the paper
| Title | Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics |
|---|---|
| Authors | Jennifer Foster and Josef van Genabith |
| Abstract | We evaluate discriminative parse reranking and parser self-training on a new English test set using four versions of the Charniak parser and a variety of parser evaluation metrics. The new test set consists of 1,000 hand-corrected British National Corpus parse trees. We directly evaluate parser output using both the Parseval and the Leaf Ancestor metrics. We also convert the hand-corrected and parser output phrase structure trees to dependency trees using a state-of-the-art functional tag labeller and constituent-to-dependency conversion tool, and then calculate label accuracy, unlabelled attachment and labelled attachment scores over the dependency structures. We find that reranking leads to a performance improvement on the new test set (albeit a modest one). We find that self-training using BNC data leads to significantly better results. However, it is not clear how effective self-training is when the training material comes from the North American News Corpus. |
| Language | |
| Topics | Parsing Systems, Corpus (creation, annotation, etc.), Evaluation methodologies |
Full paper |
Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics |
Slides |
Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics |
| Bibtex | @InProceedings{FOSTER08.774, author = {Jennifer Foster and Josef van Genabith}, title = {Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |