Summary of the paper
| Title | Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue |
|---|---|
| Authors | Susan Robinson, Antonio Roque and David Traum |
| Abstract | As conversational agents are now being developed to encounter more complex dialogue situations it is increasingly difficult to find satisfactory methods for evaluating these agents. Task-based measures are insufficient where there is no clearly defined task. While user-based evaluation methods may give a general sense of the quality of an agent's performance, they shed little light on the relative quality or success of specific features of dialogue that are necessary for system improvement. This paper examines current dialogue agent evaluation practices and motivates the need for a more detailed approach for defining and measuring the quality of dialogues between agent and user. We present a framework for evaluating the dialogue competence of artificial agents involved in complex and underspecified tasks when conversing with people. A multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent's responses to these utterances. The scheme is outlined, and then used to evaluate Staff Duty Officer Moleno, a virtual guide in Second Life. |
| Topics | Evaluation methodologies, Dialogue, Tools, systems, applications |
Full paper |
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue |
Slides |
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue |
| Bibtex | @InProceedings{ROBINSON10.674, author = {Susan Robinson and Antonio Roque and David Traum}, title = {Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |