This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication -- an aspect that is relevant to the generation of natural multimodal expressions in ECAs.
@InProceedings{PAGGIO18.12, author = {Patrizia Paggio and Costanza Navarretta}, title = {Temporal Coordination of Facial Expressions and Head Movements in First Encounter Dialogues}, booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, month = {may}, date = {7-12}, location = {Miyazaki, Japan}, editor = {Hanae Koiso and Patrizia Paggio}, publisher = {European Language Resources Association (ELRA)}, address = {Paris, France}, isbn = {979-10-95546-16-0}, language = {english} }
Summary of the paper