User-State Labeling Procedures For The Multimodal Data Collection Of SmartKom

Silke Steininger (Institute of Phonetics and Speech Communication Ludwig-Maximilians-Universität, Schellingstr.3, 80799 Munich, Germany))

Florian Schiel (Bavarian Archive for Speech Signals (BAS) Ludwig-Maximilians-Universität, Schellingstr.3, 80799 Munich, Germany)

Angelika Glesner (Institute of Phonetics and Speech Communication Ludwig-Maximilians-Universität, Schellingstr.3, 80799 Munich, Germany))

MMP1: Multimodal Resources And Tools

This contribution deals with the user-state labeling procedures of a multimodal data corpus that is created in the SmartKom project. The goal of the SmartKom project is the development of an intelligent computer-user interface that allows almost natural communication with an adaptive and self-explanatory machine. The system does not only allow input in the form of natural speech but also in the form of gestures. Additionally, facial expressions are analyzed. For the training of recognizers and the exploration of how users interact with the system, data is collected. The data comprises video and audio recordings from which the speech is transliterated and gestures and user-states are labeled. This paper gives an in depth description of the different annotation procedures for user-states. Some preliminary results will be presented, particularly a description of the homogeneity of the different user-states and their most important features.

SmartKom

352.pdf