Summary of the paper

Title Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation
Authors Verena Rieser and Oliver Lemon
Abstract The ultimate goal when building dialogue systems is to satisfy the needs of real users, but quality assurance for dialogue strategies is a non-trivial problem. The applied evaluation metrics and resulting design principles are often obscure, emerge by trial-and-error, and are highly context dependent. This paper introduces data-driven methods for obtaining reliable objective functions for system design. In particular, we test whether an objective function obtained from Wizard-of-Oz (WOZ) data is a valid estimate of real users’ preferences. We test this in a test-retest comparison between the model obtained from the WOZ study and the models obtained when testing with real users. We can show that, despite a low fit to the initial data, the objective function obtained from WOZ data makes accurate predictions for automatic dialogue evaluation, and, when automatically optimising a policy using these predictions, the improvement over a strategy simply mimicking the data becomes clear from an error analysis.
Language
Topics Acquisition, Machine Learning, Usability, user satisfaction, Dialogue & Natural Interactivity
Full paper Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation
Slides Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation
Bibtex @InProceedings{RIESER08.592,
  author = {Verena Rieser and Oliver Lemon},
  title = {Automatic Learning and Evaluation of User-Centered Objective Functions for Dialogue System Optimisation},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  month = {may},
  date = {28-30},
  address = {Marrakech, Morocco},
  editor = {Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Daniel Tapias},
  publisher = {European Language Resources Association (ELRA)},
  isbn = {2-9517408-4-0},
  note = {http://www.lrec-conf.org/proceedings/lrec2008/},
  language = {english}
  }

Powered by ELDA © 2008 ELDA/ELRA