FORM: An Extensible, Kinematically-based Gesture Annotation Scheme

Craig Martell (Linguistic Data Consortium University of Pennsylvania 3615 Market Street, Suite 200 Philadelphia, PA 19104; Department of Computer and Information Sciences University of Pennsylvania 200 S. 33rd St Philadelphia, PA 19104)

MMO2: Multimodal Lexicons & Corpora

Annotated corpora have played a critical role in speech and natural language research; and, there is an increasing interest in corpora-based research in sign language and gesture as well. As examples, consider the tools Anvil and MediaTagger. These are excellent tools which allow for multi-track annotation of videos of speakers or signers. With tools such as these, researchers can create corpora containing, for example, grammatical information, discourse structure, facial expression, and gesture. The issue, then, is not the ability to create corpora containing gesture and speech information, but the type of information captured when describing gestures. We present a non-semantic, geometrically-based annotation scheme, FORM, which allows an annotator to capture the kinematic information in a gesture just from videos of speakers. In addition, FORM stores this gestural information in Annotation Graph format—allowing for easy integration of gesture information with other types of communication information, e.g., discourse structure, parts of speech, intonation information, etc.

Annotation schemes

304.pdf