| Title |
Evaluation of Machine Learning Methods for Natural Language Processing Tasks |
| Authors |
Walter Daelemans (CNTS Language Technology Group, University of Antwerp, UIA, Universiteitsplein 1 (bldng A), B-2610 Antwerpen, Belgium) Veronique Hoste (CNTS Language Technology Group, University of Antwerp, UIA, Universiteitsplein 1 (bldng A), B-2610 Antwerpen, Belgium) |
| Session |
EO3: Written Systems Evaluation |
| Abstract |
We show that the methodology currently in use for comparing symbolic supervised learning methods applied to human language technology tasks is unreliable. We show that the interaction between algorithm parameter settings and feature selection within a single algorithm often accounts for a higher variation in results than differences between different algorithms or information sources. We illustrate this with experiments on a number of linguistic datasets. The consequences of this phenomenon are far-reaching, and we discuss possible solutions to this methodological problem. |
| Keywords |
Machine learning, Methodology |
| Full Paper |