| Title |
End-to-End Evaluation of Multimodal Dialogue Systems - can we Transfer Established Methods? |
| Authors |
Nicole Beringer (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany) Katerina Louka (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany) Victoria Penide-Lopez (BIOMAX Informatics AG.Lochhamer Str. 11, D-82152 Martinsried, Germany ) Uli Türk (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany) |
| Session |
EP1: Evaluation |
| Abstract |
The goal of this paper is to define a methodology for the end-to-end evaluation of the multimodal dialogue system SmartKom along the lines of the DARPA guidelines for spoken dialogue systems. The methodology consists of an extended framework for the evaluation of a multimodal dialogue system, evaluation metrics for its various components,and an approach to compare the user satisfaction with the system's technical performance. |
| Keywords |
Multimodality, SmartKom, Dialogue system evaluation, Evaluation methods |
| Full Paper |