Summary of the paper
| Title | Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure |
|---|---|
| Authors | Plaban Kr. Bhowmick, Anupam Basu and Pabitra Mitra |
| Abstract | The paper presents a new fuzzy agreement measure $\gamma_f$ for determining the agreement in multi-label and subjective annotation task. In this annotation framework, one data item may belong to a category or a class with a belief value denoting the degree of confidence of an annotator in assigning the data item to that category. We have provided a notion of disagreement based on the belief values provided by the annotators with respect to a category. The fuzzy agreement measure $\gamma_f$ has been proposed by defining different fuzzy agreement sets based on the distribution of difference of belief values provided by the annotators. The fuzzy agreement has been computed by studying the average agreement over all the data items and annotators. Finally, we elaborate on the computation $\gamma_f$ measure with a case study on emotion text data where a data item (sentence) may belong to more than one emotion category with varying belief values. |
| Topics | Corpus (creation, annotation, etc.), Emotion Recognition/Generation, Validation of LRs |
Full paper |
Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure |
Slides |
- |
| Bibtex | @InProceedings{BHOWMICK10.67, author = {Plaban Kr. Bhowmick and Anupam Basu and Pabitra Mitra}, title = {Determining Reliability of Subjective and Multi-label Emotion Annotation through Novel Fuzzy Agreement Measure}, booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)}, year = {2010}, month = {may}, date = {19-21}, address = {Valletta, Malta}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis and Mike Rosner and Daniel Tapias}, publisher = {European Language Resources Association (ELRA)}, isbn = {2-9517408-6-7}, language = {english} } |