Anveshan: A framework for analysis of multiple annotators' labeling behavior

Vikas Bhardwaj, Rebecca J. Passonneau, Ansaf Salleb-Aouissi, Nancy Ide

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations


Manual annotation of natural language to capture linguistic information is essential for NLP tasks involving supervised machine learning of semantic knowledge. Judgements of meaning can be more or less subjective, in which case instead of a single correct label, the labels assigned might vary among annotators based on the annotators' knowledge, age, gender, intuitions, background, and so on. We introduce a framework "Anveshan," where we investigate annotator behavior to find outliers, cluster annotators by behavior, and identify confusable labels. We also investigate the effectiveness of using trained annotators versus a larger number of untrained annotators on a word sense annotation task. The annotation data comes from a word sense disambiguation task for polysemous words, annotated by both trained annotators and untrained annotators from Amazon's Mechanical turk. Our results show that Anveshan is effective in uncovering patterns in annotator behavior, and we also show that trained annotators are superior to a larger number of untrained annotators for this task.

Original languageEnglish (US)
Title of host publicationACL 2010 - LAW 2010
Subtitle of host publication4th Linguistic Annotation Workshop, Proceedings
Number of pages9
StatePublished - 2010
Event4th Linguistic Annotation Workshop, LAW 2010 - Uppsala, Sweden
Duration: Jul 15 2010Jul 16 2010

Publication series

NameACL 2010 - LAW 2010: 4th Linguistic Annotation Workshop, Proceedings


Other4th Linguistic Annotation Workshop, LAW 2010

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Linguistics and Language


Dive into the research topics of 'Anveshan: A framework for analysis of multiple annotators' labeling behavior'. Together they form a unique fingerprint.

Cite this