LREC 2000 - Papers

LREC 2000 2^nd International Conference on Language Resources & Evaluation

Conference Papers

Papers by paper title: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Papers by ID number: 1-50, 51-100, 101-150, 151-200, 201-250, 251-300, 301-350, 351-377.

List of all papers and abstracts.

Previous Paper Next Paper

Title Predictive Performance of Dialog Systems

Authors Bonneau-Maynard H. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, hbm,devil,rossetg@limsi.fr)
Devillers L. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, devil@limsi.fr)
Rosset S. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, rosset@limsi.fr)

Keywords Dialog System, Evaluation, Performances Measures

Session Session SO2 - Dialogue Evaluation Methods

Abstract This paper relates some of our experiments on the possibility of predictive performance measures of dialog systems. Experimenting dialog systems is often a very high cost procedure due to the necessity to carry out user trials. Obviously it is advantageous when evaluation can be carried out automatically. It would be helpfull if for each application we were able to measure the system performances by an objective cost function. This performance function can be used for making predictions about a future evolution of the systems without user interaction. Using the PARADISE paradigm, a performance function derived from the relative contribution of various factors is first obtained for one system developed at LIMSI: PARIS-SITI (kiosk for tourist information retrieval in Paris). A second experiment with PARIS-SITI with a new test population confirms that the most important predictors of user satisfaction are understanding accuracy, recognition accuracy and number of user repetitions. Futhermore, similar spoken dialog features appear as important features for the Arise system (train timetable telephone information system). We also explore different ways of measuring user satisfaction. We then discuss the introduction of subjective factors in the predictive coefficients.

ana">

Title	Predictive Performance of Dialog Systems
Authors	Bonneau-Maynard H. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, hbm,devil,rossetg@limsi.fr) Devillers L. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, devil@limsi.fr) Rosset S. (LIMSI-CNRS, BP 133 91403 Orsay cedex, FRANCE, rosset@limsi.fr)
Keywords	Dialog System, Evaluation, Performances Measures
Session	Session SO2 - Dialogue Evaluation Methods
Abstract	This paper relates some of our experiments on the possibility of predictive performance measures of dialog systems. Experimenting dialog systems is often a very high cost procedure due to the necessity to carry out user trials. Obviously it is advantageous when evaluation can be carried out automatically. It would be helpfull if for each application we were able to measure the system performances by an objective cost function. This performance function can be used for making predictions about a future evolution of the systems without user interaction. Using the PARADISE paradigm, a performance function derived from the relative contribution of various factors is first obtained for one system developed at LIMSI: PARIS-SITI (kiosk for tourist information retrieval in Paris). A second experiment with PARIS-SITI with a new test population confirms that the most important predictors of user satisfaction are understanding accuracy, recognition accuracy and number of user repetitions. Futhermore, similar spoken dialog features appear as important features for the Arise system (train timetable telephone information system). We also explore different ways of measuring user satisfaction. We then discuss the introduction of subjective factors in the predictive coefficients.