Summary of the paper
| Title | Rapid creation of large-scale corpora and frequency dictionaries |
|---|---|
| Authors | Attila Zséder, Gábor Recski, Dániel Varga and András Kornai |
| Abstract | We describe, and make public, large-scale language resources and the toolchain used in their creation, for fifteen medium density European languages: Catalan, Czech, Croatian, Danish, Dutch, Finnish, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Serbian, Slovak, Spanish, and Swedish. To make the process uniform across languages, we selected tools that are either language-independent or easily customizable for each language, and reimplemented all stages that were taking too long. To achieve processing times that are insignificant compared to the time data collection (crawling) takes, we reimplemented the standard sentence- and word-level tokenizers and created new boilerplate and near-duplicate detection algorithms. Preliminary experiments with non-European languages indicate that our methods are now applicable not just to our sample, but the entire population of digitally viable languages, with the main limiting factor being the availability of high quality stemmers. |
| Topics | Corpus (creation, annotation, etc.), Endangered languages, Tools, systems, applications |
Full paper  |
Rapid creation of large-scale corpora and frequency dictionaries |
| Bibtex | @InProceedings{ZSDER12.783, author = {Attila Zséder and Gábor Recski and Dániel Varga and András Kornai}, title = {Rapid creation of large-scale corpora and frequency dictionaries}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |