Pyramid method is an existing content analysis approach in automatic summarization evaluation for manual construction of a pyramid content model from reference summaries, and manual scoring of the target summaries with the pyramid model. PyrEval assesses the content of automatic summarization by automating the manual pyramid method. PyrEval uses low-dimension distributional semantics to represent phrase meanings, and a new algorithm, EDUA (Emergent Discoveries of Units of Attractions), for solving set packing problem in construction of content model from vectorized phrases. Because the vectors are pretrained, and EDUA is an efficient greedy algorithm, PyrEval can replace manual pyramid with no retraining, and is very efficient. Moreover, PyrEval has been tested on many datasets derived from humans and machine translated summaries and shown good performance on both.
@InProceedings{GAO18.1096, author = {Yanjun Gao and Andrew Warner and Rebecca Passonneau}, title = "{PyrEval: An Automated Method for Summary Content Analysis}", booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, month = {May 7-12, 2018}, address = {Miyazaki, Japan}, editor = {Nicoletta Calzolari (Conference chair) and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Koiti Hasida and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Hélène Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis and Takenobu Tokunaga}, publisher = {European Language Resources Association (ELRA)}, isbn = {979-10-95546-00-9}, language = {english} }
Summary of the paper