Automation of summary evaluation by the pyramid method

Aaron Harnly, Ani Nenkova, Rebecca Passonneau, Owen Rambow

Research output: Chapter in Book/Report/Conference proceedingConference contribution

37 Scopus citations

Abstract

The manual Pyramid method for summary evaluation, which focuses on the task of determining if a summary expresses the same content as a set of manual models, has shown sufficient promise that the Document Understanding Conference 2005 effort will make use of it. However, an automated approach would make the method far more useful for developers and evaluators of automated summarization systems. We present an experimental environment for testing automated evaluation of summaries, pre-annotated for shared information. We reduce the problem to a combination of similarity measure computation and clustering. The best results are achieved with a unigram overlap similaritymeasure and singlelink clustering, which yields high correlation to manual pyramid scores (r=0.942, p=0.01), and shows better correlation than the n-gram overlap automatic approaches of the ROUGE system.

Original languageEnglish (US)
Title of host publicationInternational Conference on Recent Advances in Natural Language Processing, RANLP 2005 - Proceedings
EditorsGalia Angelova, Kalina Bontcheva, Ruslan Mitkov, Nicolas Nicolov, Nikolai Nikolov
PublisherAssociation for Computational Linguistics (ACL)
Pages226-232
Number of pages7
ISBN (Electronic)9549174336
StatePublished - 2005
EventInternational Conference on Recent Advances in Natural Language Processing, RANLP 2005 - Borovets, Bulgaria
Duration: Sep 21 2005Sep 23 2005

Publication series

NameInternational Conference Recent Advances in Natural Language Processing, RANLP
Volume2005-January
ISSN (Print)1313-8502

Other

OtherInternational Conference on Recent Advances in Natural Language Processing, RANLP 2005
Country/TerritoryBulgaria
CityBorovets
Period9/21/059/23/05

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Automation of summary evaluation by the pyramid method'. Together they form a unique fingerprint.

Cite this