| Title |
Sensitivity of IR systems Evaluation to Topic Difficulty |
| Authors |
Koji Eguchi (National Institute of Informatics (NII) 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan) Kazuko Kuriyama (Shirayuri College 1-25 Midorigaoka, Chofu-shi, Tokyo 182-8525, Japan) Noriko Kando (National Institute of Informatics (NII) 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan) |
| Session |
EP1: Evaluation |
| Abstract |
The difficulty of the topics or queries is one of important factors in evaluating information retrieval (IR) systems. This paper analyzes the differences of system ranking affected by the topic difficulty using a test collection ’NTCIR-1,’ which is constructed for evaluating Japanese IR systems and composed of (1) the topics, (2) the document database, and (3) the lists of relevant judgments. Furthermore, this paper defines measures for the various features on the topics, and analyzes the correlation between them, in order to investigate the predictability of the topic difficulty. |
| Keywords |
Topic, Evaluation |
| Full Paper |