Summary of the paper
| Title | Language Richness of the Web |
|---|---|
| Authors | Martin Majliš and Zdeněk Žabokrtský |
| Abstract | We have built a corpus containing texts in 106 languages from texts available on the Internet and on Wikipedia. The W2C Web Corpus contains 54.7~GB of text and the W2C Wiki Corpus contains 8.5~GB of text. The W2C Web Corpus contains more than 100~MB of text available for 75 languages. At least 10~MB of text is available for 100 languages. These corpora are a unique data source for linguists, since they outclass all published works both in the size of the material collected and the number of languages covered. This language data resource can be of use particularly to researchers specialized in multilingual technologies development. We also developed software that greatly simplifies the creation of a new text corpus for a given language, using text materials freely available on the Internet. Special attention was given to components for filtering and de-duplication that allow to keep the material quality very high. |
| Topics | Multilinguality, Corpus (creation, annotation, etc.), Language Identification |
Full paper  |
Language Richness of the Web |
| Bibtex | @InProceedings{MAJLI12.267, author = {Martin Majliš and Zdeněk Žabokrtský}, title = {Language Richness of the Web}, booktitle = {Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, month = {may}, date = {23-25}, address = {Istanbul, Turkey}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-7-7}, language = {english} } |