LREC 2000 2nd International Conference on Language Resources & Evaluation
 

Previous Paper   Next Paper

Title Galaxy-II as an Architecture for Spoken Dialogue Evaluation
Authors Polifroni Joseph (Spoken Language Systems Group, MIT Laboratory for Computer Science, 545 Technology Square, Cambridge, MA 02139 USA, joe@lcs.mit.edu)
Seneff Stephanie (Spoken Language Systems Group, MIT Laboratory for Computer Science, 545 Technology Square, Cambridge, MA 02139 USA, seneff@lcs.mit.edu)
Keywords Evaluation, Spoken Dialogue Architectures, Spoken Dialogue Dystems
Session Session SO5 - Evaluation of Dialogue
Full Paper 116.ps, 116.pdf
Abstract The GALAXY-II architecture, comprised of a centralized hub mediating the interaction among a suite of human language technology servers, provides both a useful tool for implementing systems and also a streamlined way of configuring the evaluation of these systems. In this paper, we discuss our ongoing efforts in evaluation of spoken dialogue systems, with particular attention to the way in which the architecture facilitates the development of a variety of evaluation configurations. We furthermore propose two new metrics for automatic evaluation of the discourse and dialogue components of a spoken dialogue system, which we call “user frustration” and “information bit rate.”