| LREC 2000 2nd International Conference on Language Resources & Evaluation | ||||||
|
|
|
|
|
|
|
| Title | Evaluating Multi-party Multi-modal Systems |
| Authors | Damianos Laurie E. (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA, laurie@mitre.org) Drury Jill (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA, jldrury@mitre.org) Fanderclai Tari (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA, tari@nwe.ufl.edu) Hirschman Lynette (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA, lynette@mitre.org) Oshika Beatrice (The MITRE Corporation, 202 Burlington Road;Bedford,MA 01730 USA, bea@mitre.org) |
| Keywords | Cenario, Collaborative System, Data Visualization, Evaluation Methodology, Human-Computer Interaction, Metrics, Usability |
| Session | Session SO6 - Recognition |
| Full Paper | 368.ps, 368.pdf |
| Abstract | The MITRE Corporation ’s Evaluation Working Group has developed a methodology for evaluating multi-modal groupware systems and capturing data on human-human interactions.The methodology consists of a framework for describing collaborative systems, scenario-based evaluation approach,and evaluation metrics for the various components of collaborative systems.We designed and ran two sets of experiments to validate the methodology by evaluating collaborative systems.In one experiment,we compared two configurations of a multi-modal collaborative application using a map navigation scenario requiring information sharing and decision making.In the second experiment,we pplied the evaluation methodology to a loosely integrated set of collaborative tools,again using a scenario-based approach.In both experiments,multi-modal,multi-user data were collected,visualized,annotated,and analyzed. |