TY - GEN
T1 - Automation of summary evaluation by the pyramid method
AU - Harnly, Aaron
AU - Nenkova, Ani
AU - Passonneau, Rebecca
AU - Rambow, Owen
PY - 2005
Y1 - 2005
N2 - The manual Pyramid method for summary evaluation, which focuses on the task of determining if a summary expresses the same content as a set of manual models, has shown sufficient promise that the Document Understanding Conference 2005 effort will make use of it. However, an automated approach would make the method far more useful for developers and evaluators of automated summarization systems. We present an experimental environment for testing automated evaluation of summaries, pre-annotated for shared information. We reduce the problem to a combination of similarity measure computation and clustering. The best results are achieved with a unigram overlap similaritymeasure and singlelink clustering, which yields high correlation to manual pyramid scores (r=0.942, p=0.01), and shows better correlation than the n-gram overlap automatic approaches of the ROUGE system.
AB - The manual Pyramid method for summary evaluation, which focuses on the task of determining if a summary expresses the same content as a set of manual models, has shown sufficient promise that the Document Understanding Conference 2005 effort will make use of it. However, an automated approach would make the method far more useful for developers and evaluators of automated summarization systems. We present an experimental environment for testing automated evaluation of summaries, pre-annotated for shared information. We reduce the problem to a combination of similarity measure computation and clustering. The best results are achieved with a unigram overlap similaritymeasure and singlelink clustering, which yields high correlation to manual pyramid scores (r=0.942, p=0.01), and shows better correlation than the n-gram overlap automatic approaches of the ROUGE system.
UR - http://www.scopus.com/inward/record.url?scp=84962641911&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84962641911&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:84962641911
T3 - International Conference Recent Advances in Natural Language Processing, RANLP
SP - 226
EP - 232
BT - International Conference on Recent Advances in Natural Language Processing, RANLP 2005 - Proceedings
A2 - Angelova, Galia
A2 - Bontcheva, Kalina
A2 - Mitkov, Ruslan
A2 - Nicolov, Nicolas
A2 - Nikolov, Nikolai
PB - Association for Computational Linguistics (ACL)
T2 - International Conference on Recent Advances in Natural Language Processing, RANLP 2005
Y2 - 21 September 2005 through 23 September 2005
ER -