Summary of the paper
| Title | Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation |
|---|---|
| Authors | Verena Rieser and Oliver Lemon |
| Abstract | The ultimate goal when building dialogue systems is to satisfy the needs of real users, but quality assurance for dialogue strategies is a non-trivial problem. The applied evaluation metrics and resulting design principles are often obscure, emerge by trial-and-error, and are highly context dependent. This paper introduces data-driven methods for obtaining reliable objective functions for system design. In particular, we test whether an objective function obtained from Wizard-of-Oz (WOZ) data is a valid estimate of real users’ preferences. We test this in a test-retest comparison between the model obtained from the WOZ study and the models obtained when testing with real users. We can show that, despite a low fit to the initial data, the objective function obtained from WOZ data makes accurate predictions for automatic dialogue evaluation, and, when automatically optimising a policy using these predictions, the improvement over a strategy simply mimicking the data becomes clear from an error analysis. |
| Language | |
| Topics | Acquisition, Machine Learning, Usability, user satisfaction, Dialogue & Natural Interactivity |
Full paper |
Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation |
Slides |
Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation |
| Bibtex | @InProceedings{RIESER08.592, author = {Verena Rieser and Oliver Lemon}, title = {Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, month = {may}, date = {28-30}, address = {Marrakech, Morocco}, editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-4-0}, note = {http://www.lrec-conf.org/proceedings/lrec2008/}, language = {english} } |