Summary of the paper
| Title | Estimation Method of User Satisfaction Using N-gram-based Dialog History Model for Spoken Dialog System |
|---|---|
| Authors | Sunao Hara, Norihide Kitaoka and Kazuya Takeda |
| Abstract | In this paper, we propose an estimation method of user satisfaction for a spoken dialog system using an N-gram-based dialog history model. We have collected a large amount of spoken dialog data accompanied by usability evaluation scores by users in real environments. The database is made by a field-test in which naive users used a client-server music retrieval system with a spoken dialog interface on their own PCs. An N-gram model is trained from the sequences that consist of users' dialog acts and/or the system's dialog acts for each one of six user satisfaction levels: from 1 to 5 and φ (task not completed). Then, the satisfaction level is estimated based on the N-gram likelihood. Experiments were conducted on the large real data and the results show that our proposed method achieved good classification performance; the classification accuracy was 94.7% in the experiment on a classification into dialogs with task completion and those without task completion. Even if the classifier detected all of the task incomplete dialog correctly, our proposed method achieved the false detection rate of only 6%. |
| Topics | Dialogue, Evaluation methodologies, Usability, user satisfaction |
Full paper |
Estimation Method of User Satisfaction Using N-gram-based Dialog History Model for Spoken Dialog System |
Slides |
Estimation Method of User Satisfaction Using N-gram-based Dialog History Model for Spoken Dialog System |
| Bibtex | @InProceedings{HARA10.579, author = {Sunao Hara and Norihide Kitaoka and Kazuya Takeda}, title = {Estimation Method of User Satisfaction Using N-gram-based Dialog History Model for Spoken Dialog System}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |