Summary of the paper
| Title | DICIT: Evaluation of a Distant-talking Speech Interface for Television |
|---|---|
| Authors | Timo Sowa, Fiorenza Arisio and Luca Cristoforetti |
| Abstract | The EC-funded project DICIT developed distant-talking interfaces for interactive TV. The final DICIT prototype system processes multimodal user input by speech and remote control. It was designed to understand both natural language and command-and-control-style speech input. We conducted an evaluation campaign to examine the usability and performance of the prototype. The task-oriented evaluation involved naive test persons and consisted of a subjective part with a usability questionnaire and an objective part. We used three groups of objective metrics to assess the system: one group related to speech component performance, one related to interface design and user awareness, and a final group related to task-based effectiveness and usability. These metrics were acquired with a dedicated transcription and annotation tool. The evaluation revealed a quite positive subjective assessments of the system and reasonable objective results. We report how the objective metrics helped us to determine problems in specific areas and to distinguish design-related issues from technical problems. The metrics computed over modality-specific groups also show that speech input gives a usability advantage over remote control for certain types of tasks. |
| Topics | Corpus (creation, annotation, etc.), Evaluation methodologies, Speech Recognition/Understanding |
Full paper |
DICIT: Evaluation of a Distant-talking Speech Interface for Television |
Slides |
- |
| Bibtex | @InProceedings{SOWA10.894, author = {Timo Sowa and Fiorenza Arisio and Luca Cristoforetti}, title = {DICIT: Evaluation of a Distant-talking Speech Interface for Television}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |