Evaluating content selection in summarization: The pyramid method

Ani Nenkova; Rebecca Passonneau

Evaluating content selection in summarization: The pyramid method

Ani Nenkova, Rebecca Passonneau

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

411 Scopus citations

Abstract

We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.

Original language	English (US)
Title of host publication	HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	145-152
Number of pages	8
ISBN (Electronic)	193243223X, 9781932432237
State	Published - 2004
Event	2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004 - Boston, United States Duration: May 2 2004 → May 7 2004

Publication series

Name	HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference

Conference

Conference	2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004
Country/Territory	United States
City	Boston
Period	5/2/04 → 5/7/04

All Science Journal Classification (ASJC) codes

Language and Linguistics
Linguistics and Language

Cite this

Nenkova, A., & Passonneau, R. (2004). Evaluating content selection in summarization: The pyramid method. In HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference (pp. 145-152). (HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference). Association for Computational Linguistics (ACL).

Nenkova, Ani ; Passonneau, Rebecca. / Evaluating content selection in summarization : The pyramid method. HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. Association for Computational Linguistics (ACL), 2004. pp. 145-152 (HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference).

@inproceedings{09a09291407c4287a53be97f76740898,

title = "Evaluating content selection in summarization: The pyramid method",

abstract = "We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.",

author = "Ani Nenkova and Rebecca Passonneau",

note = "Publisher Copyright: {\textcopyright} 2004 HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. All rights reserved.; 2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004 ; Conference date: 02-05-2004 Through 07-05-2004",

year = "2004",

language = "English (US)",

series = "HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "145--152",

booktitle = "HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference",

}

Nenkova, A & Passonneau, R 2004, Evaluating content selection in summarization: The pyramid method. in HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference, Association for Computational Linguistics (ACL), pp. 145-152, 2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, United States, 5/2/04.

Evaluating content selection in summarization: The pyramid method. / Nenkova, Ani; Passonneau, Rebecca.
HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. Association for Computational Linguistics (ACL), 2004. p. 145-152 (HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Evaluating content selection in summarization

T2 - 2004 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004

AU - Nenkova, Ani

AU - Passonneau, Rebecca

N1 - Publisher Copyright: © 2004 HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. All rights reserved.

PY - 2004

Y1 - 2004

N2 - We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.

AB - We present an empirically grounded method for evaluating content selection in summarization. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative importance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus improves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.

UR - http://www.scopus.com/inward/record.url?scp=85013202438&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85013202438&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85013202438

T3 - HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference

SP - 145

EP - 152

BT - HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference

PB - Association for Computational Linguistics (ACL)

Y2 - 2 May 2004 through 7 May 2004

ER -

Nenkova A, Passonneau R. Evaluating content selection in summarization: The pyramid method. In HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference. Association for Computational Linguistics (ACL). 2004. p. 145-152. (HLT-NAACL 2004 - Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference).

Evaluating content selection in summarization: The pyramid method

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this