Summary of the paper
| Title | A Trainable Tokenizer, solution for multilingual texts and compound expression tokenization |
|---|---|
| Authors | Oana Frunza |
| Abstract | Tokenization is one of the initial steps done for almost any text processing task. It is not particularly recognized as a challenging task for English monolingual systems but it rapidly increases in complexity for systems that apply it for different languages. This article proposes a supervised learning approach to perform the tokenization task. The method presented in this article is based on character transitions representation, a representation that allows compound expressions to be recognized as a single token. Compound tokens are identified independent of the character that creates the expression. The method automatically learns tokenization rules from a pre-tokenized corpus. The results obtained using the trainable system show that for Romanian and English a statistical significant improvement is obtained over a baseline system that tokenizes texts on every non-alphanumeric character. |
| Language | |
| Topics | Multilinguality, Text mining, Acquisition, Machine Learning |
Full paper |
A Trainable Tokenizer, solution for multilingual texts and compound expression tokenization |
Slides |
- |
| Bibtex | @InProceedings{FRUNZA08.152, author = {Oana Frunza}, title = {A Trainable Tokenizer, solution for multilingual texts and compound expression tokenization}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |