Summary of the paper
| Title | Evaluating Lexical Substitution: Analysis and New Measures |
|---|---|
| Authors | Sanaz Jabbari, Mark Hepple and Louise Guthrie |
| Abstract | Lexical substitution is the task of finding a replacement for a target word in a sentence so as to preserve, as closely as possible, the meaning of the original sentence. It has been proposed that lexical substitution be used as a basis for assessing the performance of word sense disambiguation systems, an idea realised in the English Lexical Substitution Task of SemEval-2007. In this paper, we examine the evaluation metrics used for the English Lexical Substitution Task and identify some problems that arise for them. We go on to propose some alternative measures for this purpose, that avoid these problems, and which in turn can be seen as redefining the key tasks that lexical substitution systems should be expected to perform. We hope that these new metrics will better serve to guide the development of lexical substitution systems in future work. One of the new metrics addresses how effective systems are in ranking substitution candidates, a key ability for lexical substitution systems, and we report some results concerning the assessment of systems produced by this measure as compared to the relevant measure from SemEval-2007. |
| Topics | Word Sense Disambiguation, Evaluation methodologies, Textual Entailment and Paraphrasing |
Full paper |
Evaluating Lexical Substitution: Analysis and New Measures |
Slides |
Evaluating Lexical Substitution: Analysis and New Measures |
| Bibtex | @InProceedings{JABBARI10.782, author = {Sanaz Jabbari and Mark Hepple and Louise Guthrie}, title = {Evaluating Lexical Substitution: Analysis and New Measures}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |