Summary of the paper
| Title | Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese |
|---|---|
| Authors | Kikuo Maekawa, Makoto Yamazaki, Takehiko Maruyama, Masaya Yamaguchi, Hideki Ogura, Wakako Kashino, Toshinobu Ogiso, Hanae Koiso and Yasuharu Den |
| Abstract | Compilation of a 100 million words balanced corpus called the Balanced Corpus of Contemporary Written Japanese (or BCCWJ) is underway at the National Institute for Japanese Language and Linguistics. The corpus covers a wide range of text genres including books, magazines, newspapers, governmental white papers, textbooks, minutes of the National Diet, internet text (bulletin board and blogs) and so forth, and when possible, samples are drawn from the rigidly defined statistical populations by means of random sampling. All texts are dually POS-analyzed based upon two different, but mutually related, definitions of ‘word.’ Currently, more than 90 million words have been sampled and XML annotated with respect to text-structure and lexical and character information. A preliminary linear discriminant analysis of text genres using the data of POS frequencies and sentence length revealed it was possible to classify the text genres with a correct identification rate of 88% as far as the samples of books, newspapers, whitepapers, and internet bulletin boards are concerned. When the samples of blogs were included in this data set, however, the identification rate went down to 68%, suggesting the considerable variance of the blog texts in terms of the textual register and style. |
| Topics | Corpus (creation, annotation, etc.), Morphology, Document Classification, Text categorisation |
Full paper |
Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese |
Slides |
- |
| Bibtex | @InProceedings{MAEKAWA10.99, author = {Kikuo Maekawa and Makoto Yamazaki and Takehiko Maruyama and Masaya Yamaguchi and Hideki Ogura and Wakako Kashino and Toshinobu Ogiso and Hanae Koiso and Yasuharu Den}, title = {Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |