| Title |
Scithon™ - An evaluation framework for assessing research productivity tools |
| Authors |
Ronin Wu, Valentin Stauber, Viktor Botev, Jacobo Elosua, Anita Brede, Maria Ritola, Kaloyan Marinov |
| Abstract |
There is a current scarcity of tested methods to evaluate the performance of artificial intelligence-based science discovery tools. Iris.ai, an international start-up developing text understanding technology and products, has developed a novel framework for per-forming such evaluation tasks. The framework, organized around live events, involves a systematic and cross-disciplinary comparison that focuses on productivity gains and takes into account user engagement. Under this format, referred to as Scithon™, event participants are asked to address, in a compressed time frame, the early stages of a research challenge put forth by a third party. Submitted results are then evaluated externally by domain experts. The logged data, including user engagement with the system, is compared against the outcome of the Scithon™. In this paper, we present in detail the full mechanics of the Scithon™and the results obtained from a series of Scithon™ competitions run since 2016, where the presented framework is used to evaluate the productivity gains of Iris.ai’s own intelligent research assistant. Initial findings show that, compared to conventional evaluation frameworks for search engines, Scithon™ is a suitable platform for benchmarking intelligent research assistants and is able to identify advantages and disadvantages of such systems in deeper detail and complexity. |
Full paper |
Scithon™ - An evaluation framework for assessing research productivity tools
|
| Bibtex |
@InProceedings{WU18.7,
author = {Ronin Wu ,Valentin Stauber ,Viktor Botev ,Jacobo Elosua ,Anita Brede ,Maria Ritola and Kaloyan Marinov},
title = {Scithon™ - An evaluation framework for assessing research productivity tools},
booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
year = {2018},
month = {may},
date = {7-12},
location = {Miyazaki, Japan},
editor = {},
publisher = {European Language Resources Association (ELRA)},
address = {Paris, France},
isbn = {979-10-95546-20-7},
language = {english}
} |
Summary of the paper