Developing Infrastructure for the Evaluation of Single and Multi-document Summarization Systems in a Cross-lingual Environment

Horacio Saggion (Department of Computer Science University of Sheffield 211 Portobello Street, Sheffield S1 4DP, England, UK)

Dragomir Radev (School of Information & Department of Electrical Engineering and Computer  Science University of Michigan 550 E. University 3080West Hall Ann Arbor, MI 48109-1092)

Simone Teufel (Computer Laboratory Cambridge University JJ Thomson Avenue Cambridge CB3 0FD, UK)

Wai Lam (Department of Systems Engineering & Engineering Management The Chinese University of Hong Kong Shatin Hong Kong)

Stephanie M. Strassel (Linguistic Data Consortium University of Pennsylvania 3615 Market Street Philadelphia, PA 19104)

Written Systems Evaluation

We describe our work on the development of Language and Evaluation Resources for the evaluation of summaries in English and Chinese. The language resources include a parallel corpus of English and Chinese texts which are translations of each other, a set of queries in both languages, clusters of documents relevants to each query, sentence relevance measures for each sentence in the document clusters, and manual multi-document summaries at different compression rates. The evaluation resources consist of metrics for measuring the content of automatic summaries against reference summaries. The framework can be used in the evaluation of extractive, non-extractive,  single and multi-document summarization. We focus on the resources developed that are made available for the research community.

Multi-Document summarization

158.pdf