Summary of the paper
| Title | A Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization |
|---|---|
| Authors | Kai Hong, John Conroy, Benoit Favre, Alex Kulesza, Hui Lin and Ani Nenkova |
| Abstract | In the period since 2004, many novel sophisticated approaches for generic multi-document summarization have been developed. Intuitive simple approaches have also been shown to perform unexpectedly well for the task. Yet it is practically impossible to compare the existing approaches directly, because systems have been evaluated on different datasets, with different evaluation measures, against different sets of comparison systems. Here we present a corpus of summaries produced by several state-of-the-art extractive summarization systems or by popular baseline systems. The inputs come from the 2004 DUC evaluation, the latest year in which generic summarization was addressed in a shared task. We use the same settings for ROUGE automatic evaluation to compare the systems directly and analyze the statistical significance of the differences in performance. We show that in terms of average scores the state-of-the-art systems appear similar but that in fact they produce very different summaries. Our corpus will facilitate future research on generic summarization and motivates the need for development of more sensitive evaluation measures and for approaches to system combination in summarization. |
| Topics | Evaluation Methodologies |
Full paper  |
A Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization |
| Bibtex | @InProceedings{HONG14.1093, author = {Kai Hong and John Conroy and Benoit Favre and Alex Kulesza and Hui Lin and Ani Nenkova}, title = {A Repository of State of the Art and Competitive Baseline Summaries for Generic News Summarization}, booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)}, year = {2014}, month = {may}, date = {26-31}, address = {Reykjavik, Iceland}, editor = {Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Hrafn Loftsson and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis}, publisher = {European Language Resources Association (ELRA)}, isbn = {978-2-9517408-8-4}, language = {english} } |