| LREC 2000 2nd International Conference on Language Resources & Evaluation | ||||||
|
|
|
|
|
|
|
| Title | A Parallel English-Japanese Query Collection for the Evaluation of On-Line Help Systems |
| Authors | Sutcliffe Richard F.E. (Department of Computer Science and Information Systems, University of Limerick, Limerick, Ireland, Richard.Sutcliffe@ul.ie) Kurohashi Sadao (Graduate School of Informatics, Kyoto University, Yoshida-honmachi, Sakyo, Kyoto 606-8501, Japan, kuro@pine.kuee.kyoto-u.ac.jp) |
| Keywords | Parallel Multilingual Evaluation Materials |
| Session | Session EP1 - Evaluation and Written Area |
| Full Paper | 248.ps, 248.pdf |
| Abstract | An experiment concerning the creation of parallel evaluation data for information retrieval is presented. A set of English queries was gathered for the domain of wordprocessing using Lotus Ami Pro. A set of Japanese queries was then created from these. The answers to the queries were elicited from eight respondents comprising four native speakers of each language. We first describe how the queries were created and the answers elicited. We then present analyses of the responses in each language. The results show a lower level of agreeement between respondents than was expected. We discuss a refinement of the elicitation process which is designed to address this problem as well as measuring the integrity of individual respondents. |