TY - GEN
T1 - Computing reliability for coreference annotation
AU - Passonneau, Rebecca J.
PY - 2004
Y1 - 2004
N2 - Coreference annotation is annotation of language corpora to indicate wlücti expressions tiave been used to co-specify ttie same discourse entity. Wlien annotations of the same data are collected from two or more coders, the reliability of the data may need to be quantified. Two obstacles have stood in the way of applying reliability metrics: incommensurate units across annotations, and lack of a convenient representation of the coding values. Given N coders and M coding units, reliability is computed from an N-by-M matrix that records the value assigned to unit Mj by coder Nj,. The solution I present accommodates a wide range of coding choices for the annotator, while preserving the same units across codings. As a consequence, it permits a straightforward application of reliability measurement. In addition, in coreference annotation, disagreements can be complete or partial so I incorporate a distance metric to scale disagreements. This method has also been applied to a quite distinct coding task, namely semantic annotation of summaries.
AB - Coreference annotation is annotation of language corpora to indicate wlücti expressions tiave been used to co-specify ttie same discourse entity. Wlien annotations of the same data are collected from two or more coders, the reliability of the data may need to be quantified. Two obstacles have stood in the way of applying reliability metrics: incommensurate units across annotations, and lack of a convenient representation of the coding values. Given N coders and M coding units, reliability is computed from an N-by-M matrix that records the value assigned to unit Mj by coder Nj,. The solution I present accommodates a wide range of coding choices for the annotator, while preserving the same units across codings. As a consequence, it permits a straightforward application of reliability measurement. In addition, in coreference annotation, disagreements can be complete or partial so I incorporate a distance metric to scale disagreements. This method has also been applied to a quite distinct coding task, namely semantic annotation of summaries.
UR - http://www.scopus.com/inward/record.url?scp=85027717964&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85027717964&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85027717964
T3 - Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004
SP - 1503
EP - 1506
BT - Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004
A2 - Xavier, Maria Francisca
A2 - Costa, Rute
A2 - Ferreira, Fatima
A2 - Lino, Maria Teresa
A2 - Silva, Raquel
PB - European Language Resources Association (ELRA)
T2 - 4th International Conference on Language Resources and Evaluation, LREC 2004
Y2 - 26 May 2004 through 28 May 2004
ER -